pipeline_core/skills_library/all/task-reliability/SKILL.md

name	description	trigger
task-reliability	Prevent tasks from silently stalling. Covers delegation patterns, failure recovery, watchdog monitoring, and session health.	When starting any task that takes more than a few tool calls, or when a background process/delegate_task might fail

Task Reliability — Anti-Stall Protocol

Core Problem

Tasks silently stop because:

1. Subagents via hermes chat -q fail with exit code 1 and nobody notices
2. delegate_task errors are returned but not acted upon
3. Background processes crash without notification
4. Session context compresses and loses track of in-progress items
5. No watchdog monitors task health between sessions

Delegation Rules

NEVER use hermes chat -q for subagents

This CLI command returns exit code 1 frequently with no useful error info. USE delegate_task instead:

delegate_task(goal="任务描述", context="需要的所有背景信息", toolsets=["terminal", "file", "web"])

Delegate Task Error Handling

When delegate_task returns:

• status: "error" → MUST retry once with simplified goal, or take over manually
• status: "interrupted" → Parent was cancelled, work is lost. Log and report to user
• status: "success" → Verify the output before telling user it's done

Batch Delegation (parallel tasks)

When you have 3+ independent subtasks, launch them in ONE delegate_task call:

delegate_task(tasks=[
    {"goal": "Task A description", "context": "full context for A", "toolsets": [...]},
    {"goal": "Task B description", "context": "full context for B", "toolsets": [...]},
    {"goal": "Task C description", "context": "full context for C", "toolsets": [...]},
])

Results format:

{
  "results": [
    {"task_index": 0, "status": "completed", "summary": "..."},
    {"task_index": 1, "status": "completed", "summary": "..."},
    {"task_index": 2, "status": "completed", "summary": "..."}
  ],
  "total_duration_seconds": 461.08
}

Rules:

• Max 3 concurrent subagents per batch (hard limit)
• Each task object must be fully self-contained
• Check EACH result individually — some may succeed while others fail
• On mixed success/failure: log failed ones, manually complete what failed

Streaming Failure Recovery

When delegate_task subagents return:

"status": "completed" but "summary": "API call failed after 3 retries: An error occurred during streaming"

This means the subagent's API call to the LLM provider failed — the subagent DID run but couldn't complete. Recovery:

1. Log the failure with details
2. Do NOT retry with delegate_task again (same provider, same failure likely)
3. Take over manually — do the work yourself in the current session
4. This is almost always a provider-side streaming issue, not a prompt issue

Subagent Prompt Design

Each delegated task prompt MUST be:

1. Self-contained (no references to "earlier in the conversation")
2. Include exact file paths, error messages, and constraints
3. Specify expected output format (e.g., "Return the absolute path of the file you created")
4. Have a clear success criterion

Cron Agent Failures (Critical)

One-shot cron workers are unreliable

Cron agents running in isolated sessions frequently fail silently:

• 2/3 cron workers never executed in a 2026-06-02 test (scheduled but no execution record)
• 1/3 executed but produced no output (ran but didn't write results)
• Root cause unclear: possibly scheduler timing, session isolation, or gateway state

Cron agents cannot call delegate_task

Cron agents run in isolated sessions that cannot spawn sub-agents via delegate_task. The isolation prevents the recursive delegation pattern.

Workaround: Use cron agents only for self-contained tasks (terminal commands, file operations). For multi-step work requiring delegation, use the main session's delegate_task or Kanban.

delegate_task gets interrupted by user input

When delegate_task is running and the user sends a message, the subagent is killed immediately and all progress is lost.

Mitigation:

• Use Kanban for long-running tasks (isolated from user input)
• Or warn user: "Task X is running, please wait"
• Never rely on delegate_task for work > 2 minutes

Cron deliver=origin fails on CLI

Cron jobs with deliver=origin fail with "no delivery target resolved for deliver=origin" when running in CLI-only sessions (no messaging gateway).

Fix: Use deliver=local for CLI sessions. Results are visible via hermes cron list and session search.

Background Process Monitoring

Pattern for long-running commands

# Start with notify_on_complete=True
terminal(command="...", background=True, notify_on_complete=True)
# System auto-notifies on completion

# For servers that never exit:
terminal(command="...", background=True)
# Then verify: terminal("curl http://localhost:8080/health")

Process health check

After starting any background process:

1. Wait 10-30 seconds (or appropriate startup time)
2. Check: process(action="poll") for output
3. Verify: health endpoint, log signal, or expected behavior
4. If no output after 2x expected startup time → process likely crashed

User Input Logging (Non-Negotiable)

Every user message must be logged to ~/input-logs/YYYYMMDD_inputs.log. This is not optional — it creates a permanent record of all user requests.

CRITICAL: User input must NEVER interrupt a running task. If a background process, delegate_task, or long-running command is active:

1. Log the input immediately
2. Note the task switch in the current task log
3. Decide whether to interleave the new request or queue it
4. The running task continues — do NOT kill or cancel it

python3 ~/.hermes/scripts/log_helper.py input "user message text"

Full logging system details are in the task-and-input-logging skill.

Task State Tracking

Before starting a task

1. Create task log: log_helper.py start "task_name" "description"
2. Add to todo list with status tracking
3. Note expected completion criteria

During task execution

1. Append progress notes to task log: log_helper.py update <path> running "progress note"
2. If switching to a different task, log why

On task completion

1. Update status: log_helper.py update <path> completed "what was done"
2. Verify the output exists and is correct
3. Remove from todo list or mark completed

On task failure

1. MUST log: log_helper.py update <path> failed "error details"
2. MUST either retry (max 2 attempts) OR explain to user why it failed
3. NEVER silently move on to another topic

Task Status Sync

When the user asks for current task status (or periodically syncs), distinguish between historical logs (what was done before) and current state (what happened today / what's happening now). The user wants real-time current state, not yesterday's summary.

Multi-repo work reconstruction (today's commits)

Reusable script: ~/.hermes/skills/task-reliability/scripts/multi-repo-scan.sh

• No args: scan commits from today
• Pass a number to scan N days back

Inline equivalent:

cd ~/repos && for d in $(find . -maxdepth 1 -type d); do
  if [ -d "$d/.git" ]; then
    commits=$(git -C "$d" log --since="$(date +%Y-%m-%dT00:00)" --oneline --all 2>/dev/null)
    if [ -n "$commits" ]; then
      echo "=== $d ==="; echo "$commits"; echo
    fi
  fi
done

Status sync checklist (all four sources):

1. Commits from today: multi-repo scan above
2. Dirty files: git status --short per repo (uncommitted changes, untracked files)
3. Unpushed commits: git log origin/main..HEAD --oneline per repo
4. Cron jobs: cronjob list — check last_status, last_delivery_error. Note: deliver=origin fails on CLI-only sessions with "no delivery target resolved" — this is expected, not a real error. Only deliver=local jobs reliably work in CLI mode. Scheduler-death signature: last_run_at=null combined with next_run_at in the PAST means the job was NEVER executed (scheduler/gateway down), not that a run failed — flag it, the job's work has silently never happened.
5. Active processes: process(action="list")
6. Todo list: todo list

What to present

• Table format: repo | commits | push status | dirty files
• Flag anything unpushed, dirty, or with cron job errors
• Keep it terse — bullet points and tables, not prose

Session Health

At session start

1. Check: process(action="list") — any zombie processes?
2. Check: recent task logs in ~/task-logs/ — any stuck in "running" for hours?
3. If session_search works, check recent sessions for unfinished tasks
4. Check for dead agent sessions (provider crash-loops): ls -lt ~/.hermes/sessions/request_dump_* | head. Dumps with reason: max_retries_exhausted / "Provider returned an empty stream" mean a session died repeatedly and the user's task in it was never executed. Confirm via session_search(session_id=...): user messages but ZERO assistant messages = dead session. Report it and re-run the orphaned task; recommend a model fallback chain.

On context compression

When [Your active task list was preserved...] appears:

1. Check each in_progress task — is it actually running or stalled?
2. If stalled for >30 minutes without progress → flag for user or retry

Watchdog Cron Job

A cron job runs every 30 minutes to:

1. Scan ~/task-logs/ for tasks stuck in "running" state > 2 hours
2. Check for background processes that exited unexpectedly
3. Report findings to user

Set up:

cronjob(action="create",
  name="task-watchdog",
  schedule="every 30m",
  prompt="Scan ~/task-logs/ for tasks stuck in 'running' state for more than 2 hours. Check process list for crashed background processes. Report any issues. If nothing wrong, stay silent.",
  script="check_stuck_tasks.sh",
  no_agent=True,
  deliver="origin")

Recovery Pattern

When discovering a stalled task:

1. Log the discovery
2. Assess: can I resume from where it left off?
3. If yes → resume and document the gap
4. If no → explain what was lost and propose restart
5. Never pretend the stall didn't happen

Multi-Task Switching Rules

Users frequently switch between multiple concurrent tasks. This is normal and expected.

When switching tasks:

1. Note the switch in the current task log: [time] User switched to task Y, paused here
2. Resume the new task with its own log entry
3. On session start, list all running tasks so the user sees what's pending

When a task fails (not just paused):

1. Log the failure: log_helper.py update <path> failed "error details"
2. Report to user immediately — the failed task stays visible in INDEX.md
3. Do NOT silently drop it. It stays in "failed" state until user decides: retry, abandon, or fix
4. User may switch to other tasks while a failed task sits — that's fine, it's tracked in INDEX.md

Key principle: visibility over serialization. Tasks can be interleaved, paused, resumed, or abandoned — but every task's state must be visible in ~/task-logs/INDEX.md. The user should never have to ask "what happened to X?"

Log Helper Script

File: scripts/log_helper.py (this skill directory)

# Log user input (call on every user message)
python3 .hermes/scripts/log_helper.py input "message text"

# Start a new task
python3 .hermes/scripts/log_helper.py start "task_name" "description"

# Update task status
python3 .hermes/scripts/log_helper.py update <log_path> running|completed|failed "note"

# List all tasks
python3 .hermes/scripts/log_helper.py list

• Input logs: ~/input-logs/YYYY-MM-DD.log
• Task logs: ~/task-logs/YYYYMMDD_HHMMSS_taskname.log
• Index: ~/task-logs/INDEX.md (Markdown table of all tasks)