feat(v3): human-in-the-loop — interactive steps, pluggable step_type registry

- New states: waiting (step/task), rejected (step)
- New tables: pipeline_human_tasks, pipeline_step_types
- New module: step_registry.py — pluggable step_type metadata
- New module: human.py — human_complete, approval_approve, approval_reject
- Executor: detects interactive step_types, creates human_tasks, enters waiting
- Reject with rollback: approval_reject(rollback_to=step) resets steps and re-runs
- API: human_task_complete, approval_approve, approval_reject, human_task_list
- API: pipeline_step_types, pipeline_register_step_type, pipeline_unregister_step_type
- Built-in interactive types: human_task, approval_gate
- Updated DDL and appcodes
This commit is contained in:
Hermes Agent 2026-06-16 11:05:45 +08:00
parent 54c98acc25
commit b9a5810d85
12 changed files with 831 additions and 59 deletions

155
README.md
View File

@ -1,14 +1,17 @@
# pipeline-service 通用产线执行引擎 # pipeline-service v3.0 — 通用产线执行引擎
## 定位 ## 定位
通用产线执行引擎模块。把 Hermes Agent 验证过的业务流程固化为可重复、可并发的产线业务环境。 通用产线执行引擎模块。把 Hermes Agent 验证过的业务流程固化为可重复、可并发的产线业务环境。
**v3.0 新增:** 人工交互步骤human_task/approval_gate、step_type 可装卸注册、多角色协作。
## 核心价值 ## 核心价值
- Hermes Agent 中用 cron/delegate/terminal 跑通的流程 → 固化为产线步骤定义 → pipeline-service 自动调度执行 - Hermes Agent 中用 cron/delegate/terminal 跑通的流程 → 固化为产线步骤定义 → pipeline-service 自动调度执行
- 一次验证,无限次自动执行 - 一次验证,无限次自动执行
- 多租户并发:同一产线,不同租户同时使用,数据完全隔离 - 多租户并发:同一产线,不同租户同时使用,数据完全隔离
- **人机协作:** 自动步骤 + 人工步骤混合执行,支持审批驳回回退
## 架构 ## 架构
@ -17,23 +20,117 @@
├── load_pipeline_service() ← 注册函数到 ServerEnv ├── load_pipeline_service() ← 注册函数到 ServerEnv
├── pipeline_submit(tenant_id, pipeline_id, owner_id, title, params) ├── 任务生命周期
├── pipeline_list(tenant_id, pipeline_id?) │ ├── pipeline_submit(tenant_id, pipeline_id, owner_id, title, params)
├── pipeline_detail(tenant_id, task_id) │ ├── pipeline_list(tenant_id, pipeline_id?)
├── pipeline_node(tenant_id, task_id, step_name, version?) │ ├── pipeline_detail(tenant_id, task_id)
├── pipeline_modify(tenant_id, task_id, updates, rerun_from) │ ├── pipeline_node(tenant_id, task_id, step_name, version?)
├── pipeline_pause(tenant_id, task_id) │ ├── pipeline_modify(tenant_id, task_id, updates, rerun_from)
├── pipeline_resume(tenant_id, task_id) │ ├── pipeline_pause(tenant_id, task_id)
├── pipeline_cancel(tenant_id, task_id) │ ├── pipeline_resume(tenant_id, task_id)
└── pipeline_register_handler(step_type, fn) ← 注册步骤处理器 │ └── pipeline_cancel(tenant_id, task_id)
├── 步骤类型注册(可装卸)
│ ├── pipeline_step_types() ← 列出所有类型
│ ├── pipeline_register_step_type(type, meta) ← 注册新类型
│ └── pipeline_unregister_step_type(type) ← 卸载类型
├── 人工交互
│ ├── human_task_complete(tenant_id, task_id, step_name, data, operator)
│ ├── approval_approve(tenant_id, task_id, step_name, reviewer, comments)
│ ├── approval_reject(tenant_id, task_id, step_name, reviewer, comments, rollback_to)
│ └── human_task_list(tenant_id?, role?, user?, status?)
└── Handler管理
└── pipeline_register_handler(step_type, fn)
``` ```
## 引擎工作原理 ## 引擎工作原理
1. **提交任务** → 读取 pipeline_steps 表的步骤定义 → 创建 pipeline_task_steps 记录 → 启动执行 1. **提交任务** → 读取 pipeline_steps 表的步骤定义 → 创建 pipeline_task_steps 记录 → 启动执行
2. **执行循环** → 解析 DAG 依赖图 → 找到可执行步骤(所有前置完成)→ 调用 handler → 存 artifact → 继续下一步 2. **执行循环** → 解析 DAG 依赖图 → 找到可执行步骤 → 判断步骤类型:
3. **多租户** → 所有查询按 tenant_id 隔离 → 同一产线多租户并发不冲突 - **自动步骤:** 调用 handler → 存 artifact → 继续下一步
4. **人工干预** → 修改节点 artifact → 创建新版本 → BFS 计算受影响步骤 → 级联重跑 - **交互步骤human_task/approval_gate** 创建 human_tasks 记录 → 步骤进入 waiting → 任务进入 waiting
3. **人工完成** → 调用 human_task_complete/approval_approve → 步骤标记完成 → 恢复执行
4. **审批驳回** → 调用 approval_reject(rollback_to=步骤名) → 回退指定步骤 → 级联重跑
5. **多租户** → 所有查询按 tenant_id 隔离
## 状态机
### 任务状态
```
submitted → running → completed
→ failed
→ paused → running (resume)
→ waiting → running (human complete)
→ cancelled
```
### 步骤状态
```
pending → running → completed
→ failed
→ skipped
→ waiting → completed (human complete)
→ rejected (approval reject)
```
## step_type 可装卸
每条产线可以注册自己的 step_type引擎按类型匹配 handler 和交互协议。
```python
# 注册一个 SDLC 产线的步骤类型
from pipeline_service import register_step_type, register_handler
# 注册自动步骤
register_step_type("code_review_auto", {
"display_name": "自动代码审查",
"category": "devops",
"is_interactive": False,
})
# 注册人工步骤
register_step_type("code_review_manual", {
"display_name": "人工代码审查",
"category": "interactive",
"is_interactive": True,
"form_schema": {
"type": "object",
"properties": {
"approved": {"type": "boolean", "title": "是否通过"},
"comments": {"type": "string", "title": "审查意见"}
}
},
"timeout_hours": 48,
"on_timeout": "escalate",
})
# 注册 handler自动步骤需要交互步骤不需要
register_handler("code_review_auto", auto_review_handler)
```
### 内置交互类型
| step_type | 用途 | 行为 |
|-----------|------|------|
| human_task | 人工填写表单/执行操作 | 步骤 waiting → 人提交 → 继续 |
| approval_gate | 审批关卡 | 步骤 waiting → 通过继续 / 驳回回退 |
### 步骤定义中的配置
在 pipeline_steps 表的 step_config JSON 中指定交互参数:
```json
{
"deps": ["develop"],
"assignee_role": "reviewer",
"assignee_id": "user123",
"form_schema": {"type": "object", "properties": {...}},
"timeout_hours": 48,
"on_timeout": "escalate"
}
```
## 数据表 ## 数据表
@ -42,25 +139,11 @@
| pipeline_tasks | 任务主表tenant_id 隔离) | | pipeline_tasks | 任务主表tenant_id 隔离) |
| pipeline_task_steps | 任务步骤执行记录 | | pipeline_task_steps | 任务步骤执行记录 |
| pipeline_artifacts | 步骤产物input/output支持版本 | | pipeline_artifacts | 步骤产物input/output支持版本 |
| **pipeline_human_tasks** | **人工任务记录v3新增** |
| **pipeline_step_types** | **步骤类型注册表v3新增** |
| pipeline_steps | 产线步骤定义(由 pipeline_core 模块管理) | | pipeline_steps | 产线步骤定义(由 pipeline_core 模块管理) |
| pipelines | 产线定义(由 pipeline_core 模块管理) | | pipelines | 产线定义(由 pipeline_core 模块管理) |
## 步骤处理器
可插拔注册,统一接口:
```python
async def handler(tenant_id, task_id, step_name, input_data, config) -> dict:
# 处理逻辑
return output_data
# 注册
from pipeline_service import register_handler
register_handler("llm_generate", handler)
```
处理器按 `step_type` 匹配。步骤定义中的 `step_type` 对应 handler 注册名。
## 宿主集成 ## 宿主集成
任何应用只需一行代码即可使用: 任何应用只需一行代码即可使用:
@ -76,7 +159,7 @@ load_pipeline_service()
- 前端交互bricks 管) - 前端交互bricks 管)
- 产线定义和定价pipeline_core/ops/dist 管) - 产线定义和定价pipeline_core/ops/dist 管)
pipeline-service 只做:调度 + 执行 + 存储。 pipeline-service 只做:调度 + 执行 + 存储 + 人工交互
## 目录结构 ## 目录结构
@ -87,18 +170,20 @@ pipeline-service/
│ ├── init.py # load_pipeline_service() + ServerEnv 注册 │ ├── init.py # load_pipeline_service() + ServerEnv 注册
│ ├── state.py # DAG 解析、步骤状态机 │ ├── state.py # DAG 解析、步骤状态机
│ ├── handler.py # 步骤处理器注册表 │ ├── handler.py # 步骤处理器注册表
│ ├── step_registry.py # 步骤类型注册表v3新增
│ ├── human.py # 人工任务操作v3新增
│ ├── storage.py # MySQL 存储层sqlor │ ├── storage.py # MySQL 存储层sqlor
│ └── executor.py # 执行循环 │ ├── executor.py # 执行循环
│ └── handlers_ktv.py # KTV产线专用 handlers
├── models/ ├── models/
│ ├── pipeline_tasks.json │ ├── pipeline_tasks.json
│ ├── pipeline_task_steps.json │ ├── pipeline_task_steps.json
│ └── pipeline_artifacts.json │ ├── pipeline_artifacts.json
│ ├── pipeline_human_tasks.json (v3新增)
│ └── pipeline_step_types.json (v3新增)
├── init/ ├── init/
│ └── data.json # appcodes 初始化数据 │ └── data.json # appcodes 初始化数据
├── scripts/
│ └── load_path.py # RBAC 权限注册
├── pyproject.toml ├── pyproject.toml
├── build.sh
└── README.md └── README.md
``` ```

View File

@ -9,7 +9,8 @@
{"k": "completed", "v": "已完成"}, {"k": "completed", "v": "已完成"},
{"k": "failed", "v": "失败"}, {"k": "failed", "v": "失败"},
{"k": "paused", "v": "已暂停"}, {"k": "paused", "v": "已暂停"},
{"k": "cancelled", "v": "已取消"} {"k": "cancelled", "v": "已取消"},
{"k": "waiting", "v": "等待人工"}
] ]
}, },
{ {
@ -20,7 +21,9 @@
{"k": "running", "v": "执行中"}, {"k": "running", "v": "执行中"},
{"k": "completed", "v": "已完成"}, {"k": "completed", "v": "已完成"},
{"k": "failed", "v": "失败"}, {"k": "failed", "v": "失败"},
{"k": "skipped", "v": "已跳过"} {"k": "skipped", "v": "已跳过"},
{"k": "waiting", "v": "等待人工"},
{"k": "rejected", "v": "已驳回"}
] ]
}, },
{ {
@ -42,7 +45,49 @@
{"k": "api_call", "v": "外部API调用"}, {"k": "api_call", "v": "外部API调用"},
{"k": "file_process", "v": "文件处理"}, {"k": "file_process", "v": "文件处理"},
{"k": "composite", "v": "合成"}, {"k": "composite", "v": "合成"},
{"k": "custom", "v": "自定义"} {"k": "custom", "v": "自定义"},
{"k": "human_task", "v": "人工任务"},
{"k": "approval_gate", "v": "审批关卡"}
]
},
{
"parentid": "human_task_type",
"parentname": "人工任务类型",
"items": [
{"k": "human_task", "v": "人工任务"},
{"k": "approval_gate", "v": "审批关卡"}
]
},
{
"parentid": "human_task_status",
"parentname": "人工任务状态",
"items": [
{"k": "pending", "v": "待处理"},
{"k": "submitted", "v": "已提交"},
{"k": "approved", "v": "已通过"},
{"k": "rejected", "v": "已驳回"},
{"k": "expired", "v": "已过期"}
]
},
{
"parentid": "step_type_category",
"parentname": "步骤类型分类",
"items": [
{"k": "media", "v": "媒体处理"},
{"k": "llm", "v": "AI生成"},
{"k": "interactive", "v": "人工交互"},
{"k": "devops", "v": "开发运维"},
{"k": "testing", "v": "测试验证"},
{"k": "general", "v": "通用"}
]
},
{
"parentid": "timeout_policy",
"parentname": "超时策略",
"items": [
{"k": "skip", "v": "跳过"},
{"k": "escalate", "v": "升级通知"},
{"k": "fail", "v": "标记失败"}
] ]
} }
] ]

View File

@ -0,0 +1,38 @@
{
"summary": [
{
"name": "pipeline_human_tasks",
"title": "人工任务表",
"primary": ["id"],
"catelog": "entity"
}
],
"fields": [
{"name": "id", "title": "主键ID", "type": "str", "length": 32, "nullable": "no"},
{"name": "task_id", "title": "产线任务ID", "type": "str", "length": 32, "nullable": "no"},
{"name": "step_name", "title": "步骤名称", "type": "str", "length": 64, "nullable": "no"},
{"name": "version", "title": "版本号", "type": "int", "nullable": "no", "default": "1"},
{"name": "task_type", "title": "任务类型", "type": "str", "length": 32, "nullable": "no"},
{"name": "assignee_role", "title": "指派角色", "type": "str", "length": 64},
{"name": "assignee_id", "title": "指派人ID", "type": "str", "length": 32},
{"name": "form_schema", "title": "表单Schema(JSON)", "type": "longtext"},
{"name": "result_data", "title": "提交结果(JSON)", "type": "longtext"},
{"name": "status", "title": "任务状态", "type": "str", "length": 32, "nullable": "no", "default": "pending"},
{"name": "submitted_by", "title": "提交人ID", "type": "str", "length": 32},
{"name": "submitted_at", "title": "提交时间", "type": "timestamp"},
{"name": "comments", "title": "备注", "type": "text"},
{"name": "created_at", "title": "创建时间", "type": "timestamp", "nullable": "no"},
{"name": "expired_at", "title": "过期时间", "type": "timestamp"}
],
"indexes": [
{"name": "idx_pht_task", "idxtype": "index", "idxfields": ["task_id"]},
{"name": "idx_pht_step", "idxtype": "index", "idxfields": ["task_id", "step_name"]},
{"name": "idx_pht_role", "idxtype": "index", "idxfields": ["assignee_role"]},
{"name": "idx_pht_assignee", "idxtype": "index", "idxfields": ["assignee_id"]},
{"name": "idx_pht_status", "idxtype": "index", "idxfields": ["status"]}
],
"codes": [
{"field": "task_type", "table": "appcodes_kv", "valuefield": "k", "textfield": "v", "cond": "parentid='human_task_type'"},
{"field": "status", "table": "appcodes_kv", "valuefield": "k", "textfield": "v", "cond": "parentid='human_task_status'"}
]
}

View File

@ -0,0 +1,31 @@
{
"summary": [
{
"name": "pipeline_step_types",
"title": "步骤类型注册表",
"primary": ["step_type"],
"catelog": "entity"
}
],
"fields": [
{"name": "step_type", "title": "步骤类型Key", "type": "str", "length": 64, "nullable": "no"},
{"name": "display_name", "title": "显示名称", "type": "str", "length": 128, "nullable": "no"},
{"name": "category", "title": "分类", "type": "str", "length": 32, "nullable": "no"},
{"name": "is_interactive", "title": "是否交互式", "type": "int", "nullable": "no", "default": "0"},
{"name": "description", "title": "描述", "type": "text"},
{"name": "form_schema", "title": "表单Schema(JSON)", "type": "longtext"},
{"name": "on_timeout", "title": "超时策略", "type": "str", "length": 16, "default": "fail"},
{"name": "timeout_hours", "title": "超时小时数", "type": "int", "default": "72"},
{"name": "pipeline_id", "title": "所属产线(NULL=全局)", "type": "str", "length": 32},
{"name": "created_at", "title": "创建时间", "type": "timestamp", "nullable": "no"},
{"name": "updated_at", "title": "更新时间", "type": "timestamp", "nullable": "no"}
],
"indexes": [
{"name": "idx_pst_category", "idxtype": "index", "idxfields": ["category"]},
{"name": "idx_pst_pipeline", "idxtype": "index", "idxfields": ["pipeline_id"]}
],
"codes": [
{"field": "category", "table": "appcodes_kv", "valuefield": "k", "textfield": "v", "cond": "parentid='step_type_category'"},
{"field": "on_timeout", "table": "appcodes_kv", "valuefield": "k", "textfield": "v", "cond": "parentid='timeout_policy'"}
]
}

View File

@ -11,12 +11,19 @@ from .init import (
pipeline_resume, pipeline_resume,
pipeline_cancel, pipeline_cancel,
pipeline_handlers, pipeline_handlers,
pipeline_step_types,
pipeline_register_step_type,
pipeline_unregister_step_type,
) )
from .handler import register_handler, list_handlers, register_default_handler from .handler import register_handler, list_handlers, register_default_handler
from .handlers_ktv import register_ktv_handlers from .handlers_ktv import register_ktv_handlers
from .step_registry import register_step_type, get_step_type, list_step_types, load_builtin_types
from .human import human_complete, approval_approve, approval_reject, human_list
from .state import ( from .state import (
STATE_PENDING, STATE_RUNNING, STATE_COMPLETED, STATE_FAILED, STATE_SKIPPED, STATE_PENDING, STATE_RUNNING, STATE_COMPLETED, STATE_FAILED, STATE_SKIPPED,
TASK_SUBMITTED, TASK_RUNNING, TASK_COMPLETED, TASK_FAILED, TASK_PAUSED, TASK_CANCELLED, STATE_WAITING, STATE_REJECTED,
TASK_SUBMITTED, TASK_RUNNING, TASK_COMPLETED, TASK_FAILED, TASK_PAUSED, TASK_CANCELLED, TASK_WAITING,
HUMAN_PENDING, HUMAN_SUBMITTED, HUMAN_APPROVED, HUMAN_REJECTED, HUMAN_EXPIRED,
) )
__version__ = "2.0.0" __version__ = "3.0.0"

View File

@ -1,22 +1,28 @@
"""Pipeline execution engine - schedules and runs steps.""" """Pipeline execution engine - schedules and runs steps.
Supports both automatic steps (machine handlers) and interactive steps
(human_task, approval_gate) that pause execution waiting for human input.
"""
import asyncio import asyncio
import json
import logging import logging
import traceback import traceback
from typing import Dict from typing import Dict
from .state import ( from .state import (
STATE_PENDING, STATE_RUNNING, STATE_COMPLETED, STATE_FAILED, STATE_PENDING, STATE_RUNNING, STATE_COMPLETED, STATE_FAILED, STATE_WAITING,
TASK_RUNNING, TASK_COMPLETED, TASK_FAILED, TASK_PAUSED, TASK_RUNNING, TASK_COMPLETED, TASK_FAILED, TASK_PAUSED, TASK_WAITING,
build_step_graph, find_next_step, build_step_graph, find_next_step, has_waiting_steps,
check_all_completed, check_any_failed, check_all_completed, check_any_failed,
) )
from .storage import ( from .storage import (
get_pipeline_steps, get_step_states, get_pipeline_steps, get_step_states,
update_task_state, update_step_state, update_task_state, update_step_state,
save_artifact, get_artifact, save_artifact, get_artifact, create_human_task,
) )
from .handler import get_handler from .handler import get_handler
from .step_registry import is_interactive, get_step_type
logger = logging.getLogger("pipeline.executor") logger = logging.getLogger("pipeline.executor")
@ -32,7 +38,7 @@ async def start_task(task_id: str):
async def resume_task(task_id: str): async def resume_task(task_id: str):
"""Resume a paused task.""" """Resume a paused or waiting task."""
task = asyncio.create_task(_run_task(task_id)) task = asyncio.create_task(_run_task(task_id))
_active_tasks[task_id] = task _active_tasks[task_id] = task
return task return task
@ -100,10 +106,16 @@ async def _run_task(task_id: str):
logger.warning(f"Task {task_id} failed (step failure)") logger.warning(f"Task {task_id} failed (step failure)")
break break
# Check if any step is waiting for human input
if has_waiting_steps(step_states):
await update_task_state(task_id, TASK_WAITING)
logger.info(f"Task {task_id} waiting for human input")
break
# Find next executable step # Find next executable step
next_step = find_next_step(step_graph, step_states) next_step = find_next_step(step_graph, step_states)
if not next_step: if not next_step:
# No executable step but not all completed - deadlock or waiting # No executable step and no waiting steps - deadlock
logger.warning(f"Task {task_id}: no executable step, states={step_states}") logger.warning(f"Task {task_id}: no executable step, states={step_states}")
await update_task_state(task_id, TASK_FAILED) await update_task_state(task_id, TASK_FAILED)
break break
@ -134,7 +146,16 @@ async def _get_task_raw(task_id: str) -> dict:
async def _execute_step(task_id: str, step_name: str, step_graph: dict, task_info: dict): async def _execute_step(task_id: str, step_name: str, step_graph: dict, task_info: dict):
"""Execute a single step.""" """Execute a single step.
For interactive step_types (human_task, approval_gate):
- Creates a human_tasks record
- Sets step to WAITING state
- Execution loop will detect waiting and pause the task
For automatic step_types:
- Calls handler, saves artifact, marks completed
"""
step_info = step_graph[step_name] step_info = step_graph[step_name]
step_type = step_info["step_type"] step_type = step_info["step_type"]
version = task_info.get('current_version', task_info.get('current_Version', 1)) version = task_info.get('current_version', task_info.get('current_Version', 1))
@ -149,6 +170,13 @@ async def _execute_step(task_id: str, step_name: str, step_graph: dict, task_inf
# Save input artifact # Save input artifact
await save_artifact(task_id, version, step_name, "input", input_data) await save_artifact(task_id, version, step_name, "input", input_data)
# Check if this is an interactive step type
if is_interactive(step_type):
await _handle_interactive_step(
task_id, step_name, step_type, version, input_data, step_info
)
return
# Look up handler by step_type # Look up handler by step_type
handler = get_handler(step_type) handler = get_handler(step_type)
if not handler: if not handler:
@ -156,8 +184,16 @@ async def _execute_step(task_id: str, step_name: str, step_graph: dict, task_inf
if not handler: if not handler:
raise ValueError(f"No handler for step_type '{step_type}' and no default handler") raise ValueError(f"No handler for step_type '{step_type}' and no default handler")
# Load step config for handler
step_config = step_info.get("step_config", {})
if isinstance(step_config, str):
try:
step_config = json.loads(step_config)
except (json.JSONDecodeError, TypeError):
step_config = {}
# Execute handler # Execute handler
output_data = await handler(tenant_id, task_id, step_name, input_data, {}) output_data = await handler(tenant_id, task_id, step_name, input_data, step_config)
# Save output artifact # Save output artifact
await save_artifact(task_id, version, step_name, "output", output_data) await save_artifact(task_id, version, step_name, "output", output_data)
@ -170,6 +206,41 @@ async def _execute_step(task_id: str, step_name: str, step_graph: dict, task_inf
await update_step_state(task_id, step_name, STATE_FAILED, error_msg) await update_step_state(task_id, step_name, STATE_FAILED, error_msg)
async def _handle_interactive_step(task_id, step_name, step_type, version, input_data, step_info):
"""Handle an interactive step — create human task record and enter WAITING."""
# Get step type metadata
meta = get_step_type(step_type) or {}
# Extract assignment info from step_config
step_config = step_info.get("step_config", {})
if isinstance(step_config, str):
try:
step_config = json.loads(step_config)
except (json.JSONDecodeError, TypeError):
step_config = {}
assignee_role = step_config.get("assignee_role", "")
assignee_id = step_config.get("assignee_id", "")
form_schema = step_config.get("form_schema") or meta.get("form_schema")
timeout_hours = step_config.get("timeout_hours") or meta.get("timeout_hours")
# Create human task record
await create_human_task(
task_id=task_id,
step_name=step_name,
version=version,
task_type=step_type,
assignee_role=assignee_role,
assignee_id=assignee_id,
form_schema=form_schema,
timeout_hours=timeout_hours,
)
# Set step to waiting
await update_step_state(task_id, step_name, STATE_WAITING)
logger.info(f"Step {step_name} waiting for human input (type={step_type}, role={assignee_role})")
async def _gather_inputs(task_id: str, version: int, deps: list) -> dict: async def _gather_inputs(task_id: str, version: int, deps: list) -> dict:
"""Gather input data from dependency step outputs.""" """Gather input data from dependency step outputs."""
inputs = {} inputs = {}

239
pipeline_service/human.py Normal file
View File

@ -0,0 +1,239 @@
"""Human task operations — complete, reject, approve.
When the executor encounters an interactive step_type (human_task/approval_gate),
it creates a pipeline_human_tasks record and puts the step into WAITING state.
This module handles the human-side operations to resume execution.
"""
import json
import logging
from appPublic.uniqueID import getID
from .state import (
STATE_WAITING, STATE_COMPLETED, STATE_REJECTED, STATE_PENDING,
TASK_RUNNING, TASK_WAITING, TASK_FAILED,
HUMAN_PENDING, HUMAN_SUBMITTED, HUMAN_APPROVED, HUMAN_REJECTED,
build_step_graph, find_next_step, get_rerun_from_next,
)
from .storage import (
get_task, get_step_states, get_pipeline_steps,
update_step_state, update_task_state, save_artifact,
reset_steps,
)
from .executor import resume_task
logger = logging.getLogger("pipeline.human")
async def human_complete(tenant_id, task_id, step_name, result_data, operator_id=None):
"""Complete a human_task step — submit form data and resume execution.
Args:
tenant_id: tenant
task_id: task
step_name: the waiting step
result_data: dict form submission data
operator_id: who completed it
Returns:
JSON string
"""
result = {"success": False}
try:
task = await get_task(tenant_id, task_id)
if not task:
result["message"] = "任务不存在"
return json.dumps(result, ensure_ascii=False)
# Verify step is waiting
step_states = await get_step_states(task_id)
if step_states.get(step_name) != STATE_WAITING:
result["message"] = f"步骤 {step_name} 不在等待状态 (当前: {step_states.get(step_name)})"
return json.dumps(result, ensure_ascii=False)
# Save result as output artifact
version = task.get("current_version", task.get("current_Version", 1))
if isinstance(version, str):
version = int(version)
await save_artifact(task_id, version, step_name, "output", result_data)
# Mark step completed
await update_step_state(task_id, step_name, STATE_COMPLETED)
# Update human_tasks record
await _update_human_task(task_id, step_name, HUMAN_SUBMITTED, result_data, operator_id)
# Resume execution
await update_task_state(task_id, TASK_RUNNING)
await resume_task(task_id)
result["success"] = True
result["message"] = f"步骤 {step_name} 已完成,产线继续执行"
except Exception as e:
result["message"] = str(e)
return json.dumps(result, ensure_ascii=False, default=str)
async def approval_approve(tenant_id, task_id, step_name, reviewer_id, comments=None):
"""Approve an approval_gate step.
Args:
reviewer_id: who approved
comments: optional comments
Returns:
JSON string
"""
result = {"success": False}
try:
task = await get_task(tenant_id, task_id)
if not task:
result["message"] = "任务不存在"
return json.dumps(result, ensure_ascii=False)
step_states = await get_step_states(task_id)
if step_states.get(step_name) != STATE_WAITING:
result["message"] = f"步骤 {step_name} 不在等待状态"
return json.dumps(result, ensure_ascii=False)
# Save approval as artifact
version = task.get("current_version", task.get("current_Version", 1))
if isinstance(version, str):
version = int(version)
approval_data = {"approved": True, "reviewer_id": reviewer_id, "comments": comments}
await save_artifact(task_id, version, step_name, "output", approval_data)
# Mark step completed
await update_step_state(task_id, step_name, STATE_COMPLETED)
await _update_human_task(task_id, step_name, HUMAN_APPROVED, approval_data, reviewer_id)
# Resume execution
await update_task_state(task_id, TASK_RUNNING)
await resume_task(task_id)
result["success"] = True
result["message"] = f"步骤 {step_name} 审批通过,产线继续执行"
except Exception as e:
result["message"] = str(e)
return json.dumps(result, ensure_ascii=False, default=str)
async def approval_reject(tenant_id, task_id, step_name, reviewer_id, comments=None, rollback_to=None):
"""Reject an approval_gate step — optionally rollback to a previous step.
Args:
reviewer_id: who rejected
comments: rejection reason
rollback_to: step_name to rollback to (if None, just mark rejected)
Returns:
JSON string
"""
result = {"success": False}
try:
task = await get_task(tenant_id, task_id)
if not task:
result["message"] = "任务不存在"
return json.dumps(result, ensure_ascii=False)
step_states = await get_step_states(task_id)
if step_states.get(step_name) != STATE_WAITING:
result["message"] = f"步骤 {step_name} 不在等待状态"
return json.dumps(result, ensure_ascii=False)
rejection_data = {"approved": False, "reviewer_id": reviewer_id, "comments": comments}
if rollback_to:
# Rollback: reset target step and all steps between
pipeline_id = task.get("pipeline_id", task.get("Pipeline_id", ""))
step_records = await get_pipeline_steps(pipeline_id)
step_graph = build_step_graph(step_records)
# Get all steps from rollback_to to step_name (inclusive)
affected = _get_steps_between(step_graph, rollback_to, step_name)
if not affected:
result["message"] = f"无法计算从 {rollback_to}{step_name} 的回退路径"
return json.dumps(result, ensure_ascii=False)
# Reset affected steps to pending
await reset_steps(task_id, affected)
# Save rejection info as artifact on the rollback target
version = task.get("current_version", task.get("current_Version", 1))
if isinstance(version, str):
version = int(version)
await save_artifact(task_id, version, rollback_to, "input", {
"__rejection__": rejection_data
})
# Resume from rollback point
await update_task_state(task_id, TASK_RUNNING)
await resume_task(task_id)
result["success"] = True
result["message"] = f"审批驳回,回退到 {rollback_to},重跑 {len(affected)} 个步骤"
result["rerun_steps"] = affected
else:
# Just reject — task fails
await update_step_state(task_id, step_name, STATE_REJECTED, comments)
await _update_human_task(task_id, step_name, HUMAN_REJECTED, rejection_data, reviewer_id)
await update_task_state(task_id, TASK_FAILED)
result["success"] = True
result["message"] = f"审批驳回,任务已标记失败"
except Exception as e:
result["message"] = str(e)
return json.dumps(result, ensure_ascii=False, default=str)
async def human_list(tenant_id=None, assignee_role=None, assignee_id=None, status=None):
"""List human tasks, optionally filtered by role/user/status.
Returns:
JSON string with human_tasks list
"""
result = {"success": False}
try:
from .storage import list_human_tasks
tasks = await list_human_tasks(tenant_id, assignee_role, assignee_id, status)
result["success"] = True
result["tasks"] = tasks
result["total"] = len(tasks)
except Exception as e:
result["message"] = str(e)
return json.dumps(result, ensure_ascii=False, default=str)
def _get_steps_between(step_graph: dict, from_step: str, to_step: str) -> list:
"""BFS from from_step to to_step through dependents. Returns ordered list including both endpoints."""
if from_step not in step_graph or to_step not in step_graph:
return []
visited = set()
queue = [from_step]
found = False
while queue:
current = queue.pop(0)
if current in visited:
continue
visited.add(current)
if current == to_step:
found = True
break
for dep in step_graph.get(current, {}).get("dependents", []):
if dep not in visited:
queue.append(dep)
if not found:
return []
result = sorted(visited, key=lambda s: step_graph.get(s, {}).get("order", 999))
return result
async def _update_human_task(task_id, step_name, status, result_data, operator_id=None):
"""Update the human_tasks record for this step."""
from .storage import update_human_task_record
await update_human_task_record(task_id, step_name, status, result_data, operator_id)

View File

@ -2,6 +2,7 @@
Hermes Agent 验证过的业务流程固化为可重复可并发的产线业务环境 Hermes Agent 验证过的业务流程固化为可重复可并发的产线业务环境
支持多租户隔离DAG 步骤调度可插拔步骤处理器artifact 版本管理 支持多租户隔离DAG 步骤调度可插拔步骤处理器artifact 版本管理
支持人工交互步骤human_task/approval_gate人机协作产线
任何宿主应用都可以通过 load_pipeline_service() 加载本模块 任何宿主应用都可以通过 load_pipeline_service() 加载本模块
""" """
@ -12,7 +13,7 @@ from appPublic.uniqueID import getID
from appPublic.log import debug from appPublic.log import debug
from .state import ( from .state import (
TASK_SUBMITTED, TASK_RUNNING, TASK_COMPLETED, TASK_FAILED, TASK_PAUSED, TASK_CANCELLED, TASK_SUBMITTED, TASK_RUNNING, TASK_COMPLETED, TASK_FAILED, TASK_PAUSED, TASK_CANCELLED, TASK_WAITING,
build_step_graph, get_cascade_rerun_steps, get_rerun_from_next, build_step_graph, get_cascade_rerun_steps, get_rerun_from_next,
) )
from .storage import ( from .storage import (
@ -20,13 +21,19 @@ from .storage import (
get_artifact, get_all_artifacts, list_tasks, get_artifact, get_all_artifacts, list_tasks,
update_task_state, update_task_version, update_task_state, update_task_version,
get_pipeline_steps, reset_steps, save_artifact, get_pipeline_steps, reset_steps, save_artifact,
get_human_task,
) )
from .executor import start_task, resume_task, stop_task, is_running from .executor import start_task, resume_task, stop_task, is_running
from .handler import register_handler, list_handlers, register_default_handler from .handler import register_handler, list_handlers, register_default_handler
from .step_registry import (
register_step_type, get_step_type, list_step_types,
unregister_step_type, load_builtin_types,
)
from .human import human_complete, approval_approve, approval_reject, human_list
from .handlers_ktv import register_ktv_handlers from .handlers_ktv import register_ktv_handlers
MODULE_NAME = "pipeline_service" MODULE_NAME = "pipeline_service"
MODULE_VERSION = "2.0.0" MODULE_VERSION = "3.0.0"
async def pipeline_submit(tenant_id, pipeline_id, owner_id, title, params=None): async def pipeline_submit(tenant_id, pipeline_id, owner_id, title, params=None):
@ -94,6 +101,14 @@ async def pipeline_detail(tenant_id, task_id):
return json.dumps(result, ensure_ascii=False) return json.dumps(result, ensure_ascii=False)
steps = await get_task_steps(task_id) steps = await get_task_steps(task_id)
# Enrich steps with human task info for interactive steps
for step in steps:
if step.get('state') == 'waiting':
ht = await get_human_task(task_id, step['step_name'])
if ht:
step['human_task'] = ht
task["steps"] = steps task["steps"] = steps
task["is_running"] = is_running(task_id) task["is_running"] = is_running(task_id)
@ -258,6 +273,23 @@ def pipeline_handlers():
return json.dumps(list_handlers(), ensure_ascii=False) return json.dumps(list_handlers(), ensure_ascii=False)
def pipeline_step_types():
"""查看所有注册的步骤类型(含元数据)。"""
return json.dumps(list_step_types(), ensure_ascii=False)
def pipeline_register_step_type(step_type, metadata):
"""注册步骤类型(可装卸)。"""
register_step_type(step_type, metadata)
return json.dumps({"success": True, "step_type": step_type}, ensure_ascii=False)
def pipeline_unregister_step_type(step_type):
"""卸载步骤类型。"""
unregister_step_type(step_type)
return json.dumps({"success": True, "step_type": step_type}, ensure_ascii=False)
def load_pipeline_service(): def load_pipeline_service():
"""注册所有函数到 ServerEnv。任何宿主应用调用此函数即可使用产线引擎。""" """注册所有函数到 ServerEnv。任何宿主应用调用此函数即可使用产线引擎。"""
env = ServerEnv() env = ServerEnv()
@ -276,11 +308,25 @@ def load_pipeline_service():
env.pipeline_register_handler = register_handler env.pipeline_register_handler = register_handler
env.pipeline_handlers = pipeline_handlers env.pipeline_handlers = pipeline_handlers
# Step type registry (pluggable)
env.pipeline_step_types = pipeline_step_types
env.pipeline_register_step_type = pipeline_register_step_type
env.pipeline_unregister_step_type = pipeline_unregister_step_type
# Human task operations
env.human_task_complete = human_complete
env.approval_approve = approval_approve
env.approval_reject = approval_reject
env.human_task_list = human_list
# Register default handler # Register default handler
register_default_handler() register_default_handler()
# Load built-in interactive step types
load_builtin_types()
# Register KTV handlers # Register KTV handlers
register_ktv_handlers() register_ktv_handlers()
debug(f"[{MODULE_NAME}] v{MODULE_VERSION} loaded — generic pipeline execution engine") debug(f"[{MODULE_NAME}] v{MODULE_VERSION} loaded — pipeline engine with human-in-the-loop support")
return True return True

View File

@ -9,6 +9,8 @@ STATE_RUNNING = "running"
STATE_COMPLETED = "completed" STATE_COMPLETED = "completed"
STATE_FAILED = "failed" STATE_FAILED = "failed"
STATE_SKIPPED = "skipped" STATE_SKIPPED = "skipped"
STATE_WAITING = "waiting" # 等待人工输入
STATE_REJECTED = "rejected" # 审批驳回
# Pipeline task states # Pipeline task states
TASK_SUBMITTED = "submitted" TASK_SUBMITTED = "submitted"
@ -17,6 +19,14 @@ TASK_COMPLETED = "completed"
TASK_FAILED = "failed" TASK_FAILED = "failed"
TASK_PAUSED = "paused" TASK_PAUSED = "paused"
TASK_CANCELLED = "cancelled" TASK_CANCELLED = "cancelled"
TASK_WAITING = "waiting" # 等待人工步骤完成
# Human task statuses
HUMAN_PENDING = "pending"
HUMAN_SUBMITTED = "submitted"
HUMAN_APPROVED = "approved"
HUMAN_REJECTED = "rejected"
HUMAN_EXPIRED = "expired"
def build_step_graph(step_records: list) -> Dict[str, dict]: def build_step_graph(step_records: list) -> Dict[str, dict]:
@ -57,21 +67,24 @@ def build_step_graph(step_records: list) -> Dict[str, dict]:
def find_next_step(step_graph: Dict[str, dict], step_states: Dict[str, str]) -> Optional[str]: def find_next_step(step_graph: Dict[str, dict], step_states: Dict[str, str]) -> Optional[str]:
"""Find the next step to execute: pending + all deps completed. """Find the next step to execute: pending + all deps completed/skipped.
Args: Args:
step_graph: from build_step_graph() step_graph: from build_step_graph()
step_states: {step_name: current_state} step_states: {step_name: current_state}
Returns: Returns:
step_name or None if all done / blocked step_name or None if all done / blocked / waiting
""" """
# Terminal states that count as "done" for dependency resolution
done_states = {STATE_COMPLETED, STATE_SKIPPED}
candidates = [] candidates = []
for name, info in step_graph.items(): for name, info in step_graph.items():
if step_states.get(name) != STATE_PENDING: if step_states.get(name) != STATE_PENDING:
continue continue
deps_ok = all( deps_ok = all(
step_states.get(dep) == STATE_COMPLETED step_states.get(dep) in done_states
for dep in info["deps"] for dep in info["deps"]
) )
if deps_ok: if deps_ok:
@ -85,6 +98,16 @@ def find_next_step(step_graph: Dict[str, dict], step_states: Dict[str, str]) ->
return candidates[0] return candidates[0]
def has_waiting_steps(step_states: Dict[str, str]) -> bool:
"""Check if any step is in waiting state."""
return any(s == STATE_WAITING for s in step_states.values())
def has_rejected_steps(step_states: Dict[str, str]) -> bool:
"""Check if any step has been rejected."""
return any(s == STATE_REJECTED for s in step_states.values())
def get_cascade_rerun_steps(step_graph: Dict[str, dict], from_step: str) -> List[str]: def get_cascade_rerun_steps(step_graph: Dict[str, dict], from_step: str) -> List[str]:
"""BFS from modified step through dependents. Returns ordered list.""" """BFS from modified step through dependents. Returns ordered list."""
if from_step not in step_graph: if from_step not in step_graph:
@ -132,5 +155,5 @@ def check_all_completed(step_states: Dict[str, str]) -> bool:
def check_any_failed(step_states: Dict[str, str]) -> bool: def check_any_failed(step_states: Dict[str, str]) -> bool:
"""Check if any step has failed.""" """Check if any step has failed or been rejected."""
return any(s == STATE_FAILED for s in step_states.values()) return any(s in (STATE_FAILED, STATE_REJECTED) for s in step_states.values())

View File

@ -0,0 +1,81 @@
"""Step type registry — pluggable step_type metadata.
Each pipeline can define its own step_types. The registry tracks:
- handler function (already in handler.py)
- metadata: display_name, category, is_interactive, form_schema, on_timeout
Built-in interactive types: human_task, approval_gate
"""
import logging
from typing import Dict, Optional
logger = logging.getLogger("pipeline.step_registry")
# step_type -> metadata dict
_REGISTRY: Dict[str, dict] = {}
# Built-in interactive step types
BUILTIN_INTERACTIVE = {
"human_task": {
"display_name": "人工任务",
"category": "interactive",
"is_interactive": True,
"description": "需要人工填写表单或执行操作后继续",
},
"approval_gate": {
"display_name": "审批关卡",
"category": "interactive",
"is_interactive": True,
"description": "需要审批人通过后继续,可驳回",
},
}
def register_step_type(step_type: str, metadata: dict):
"""Register a step type with metadata.
Args:
step_type: unique key matching pipeline_steps.step_type
metadata: dict with keys:
- display_name (str): 显示名称
- category (str): 分类 (media/llm/interactive/devops/...)
- is_interactive (bool): 是否需要人工介入, default False
- description (str): 描述
- form_schema (dict): 人工任务表单JSON Schema (可选)
- on_timeout (str): 超时策略 skip/escalate/fail (可选)
- timeout_hours (int): 超时小时数 (可选)
"""
existing = _REGISTRY.get(step_type, {})
existing.update(metadata)
_REGISTRY[step_type] = existing
logger.info(f"Registered step_type: {step_type} (interactive={metadata.get('is_interactive', False)})")
def get_step_type(step_type: str) -> Optional[dict]:
"""Get step type metadata. Returns None if not registered."""
return _REGISTRY.get(step_type)
def is_interactive(step_type: str) -> bool:
"""Check if a step type requires human interaction."""
meta = _REGISTRY.get(step_type, {})
return meta.get("is_interactive", False)
def list_step_types() -> Dict[str, dict]:
"""List all registered step types with metadata."""
return dict(_REGISTRY)
def unregister_step_type(step_type: str):
"""Remove a step type from registry."""
removed = _REGISTRY.pop(step_type, None)
if removed:
logger.info(f"Unregistered step_type: {step_type}")
def load_builtin_types():
"""Load built-in interactive step types."""
for st, meta in BUILTIN_INTERACTIVE.items():
register_step_type(st, meta)

View File

@ -234,3 +234,109 @@ async def reset_steps(task_id: str, step_names: list):
"id": rec_id, "state": "pending", "id": rec_id, "state": "pending",
"error_msg": None, "started_at": None, "completed_at": None "error_msg": None, "started_at": None, "completed_at": None
}) })
# ── Human tasks storage ──
async def create_human_task(task_id, step_name, version, task_type, assignee_role=None,
assignee_id=None, form_schema=None, timeout_hours=None):
"""Create a pipeline_human_tasks record."""
db, dbname = _get_db()
async with db.sqlorContext(dbname) as sor:
data = {
"id": getID(),
"task_id": task_id,
"step_name": step_name,
"version": version,
"task_type": task_type,
"assignee_role": assignee_role or "",
"assignee_id": assignee_id or "",
"form_schema": json.dumps(form_schema, ensure_ascii=False) if form_schema else None,
"status": "pending",
}
if timeout_hours:
from datetime import datetime, timedelta
expired = datetime.now() + timedelta(hours=timeout_hours)
data["expired_at"] = expired.strftime("%Y-%m-%d %H:%M:%S")
await sor.C('pipeline_human_tasks', data)
return data["id"]
async def update_human_task_record(task_id, step_name, status, result_data=None, operator_id=None):
"""Update a human_tasks record."""
db, dbname = _get_db()
async with db.sqlorContext(dbname) as sor:
recs = await sor.R('pipeline_human_tasks', {'task_id': task_id, 'step_name': step_name})
if not recs:
return
rec = recs[0]
rec_id = rec.id if hasattr(rec, 'id') else rec['id']
data = {"id": rec_id, "status": status}
if result_data is not None:
data["result_data"] = json.dumps(result_data, ensure_ascii=False, default=str)
if operator_id:
data["submitted_by"] = operator_id
data["submitted_at"] = "NOW()"
await sor.U('pipeline_human_tasks', data)
async def list_human_tasks(tenant_id=None, assignee_role=None, assignee_id=None, status=None):
"""List human tasks with optional filters."""
db, dbname = _get_db()
async with db.sqlorContext(dbname) as sor:
conditions = []
params = {}
if assignee_role:
conditions.append("assignee_role=${assignee_role}$")
params["assignee_role"] = assignee_role
if assignee_id:
conditions.append("assignee_id=${assignee_id}$")
params["assignee_id"] = assignee_id
if status:
conditions.append("status=${status}$")
params["status"] = status
if tenant_id:
# Join with pipeline_tasks to filter by tenant
conditions.append("task_id IN (SELECT id FROM pipeline_tasks WHERE tenant_id=${tenant_id}$)")
params["tenant_id"] = tenant_id
where = " AND ".join(conditions) if conditions else "1=1"
sql = f"SELECT * FROM pipeline_human_tasks WHERE {where} ORDER BY created_at DESC"
recs = await sor.sqlExe(sql, params)
result = []
for rec in (recs or []):
if hasattr(rec, '__dict__'):
d = {k: getattr(rec, k) for k in dir(rec) if not k.startswith('_')}
else:
d = dict(rec)
# Parse JSON fields
for field in ('form_schema', 'result_data'):
if d.get(field) and isinstance(d[field], str):
try:
d[field] = json.loads(d[field])
except (json.JSONDecodeError, TypeError):
pass
result.append(d)
return result
async def get_human_task(task_id, step_name):
"""Get a specific human task record."""
db, dbname = _get_db()
async with db.sqlorContext(dbname) as sor:
recs = await sor.R('pipeline_human_tasks', {'task_id': task_id, 'step_name': step_name})
if not recs:
return None
rec = recs[0]
if hasattr(rec, '__dict__'):
d = {k: getattr(rec, k) for k in dir(rec) if not k.startswith('_')}
else:
d = dict(rec)
for field in ('form_schema', 'result_data'):
if d.get(field) and isinstance(d[field], str):
try:
d[field] = json.loads(d[field])
except (json.JSONDecodeError, TypeError):
pass
return d

View File

@ -4,8 +4,8 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "pipeline_service" name = "pipeline_service"
version = "2.0.0" version = "3.0.0"
description = "通用产线执行引擎模块 — DAG调度、多租户隔离、可插拔步骤处理器、artifact版本管理" description = "通用产线执行引擎模块 — DAG调度、多租户隔离、可插拔步骤处理器、人工交互步骤、artifact版本管理"
requires-python = ">=3.8" requires-python = ">=3.8"
dependencies = [ dependencies = [
"sqlor", "sqlor",