pipeline_core/skill/SKILL.md

name	description
pipeline-core	Pipeline core module — pipeline definition, step orchestration, version control, and publishing.

pipeline_core

Core module for managing pipeline definitions, step DAG orchestration, version snapshots, and publishing workflow.

Directory Layout

pipeline_core/
├── pipeline_core/       # Python package
│   ├── __init__.py
│   └── init.py          # ServerEnv registration via load_pipeline_core()
├── wwwroot/
│   ├── index.ui         # Module entry — pipeline list dashboard
│   ├── api/
│   │   ├── pipelines_*.dspy      # CRUD wrappers for pipelines table
│   │   ├── pipeline_steps_*.dspy # CRUD wrappers for pipeline_steps
│   │   ├── pipeline_versions_*.dspy
│   │   └── pipeline_publish.dspy # Version publish + activate
│   └── pipeline_editor/
│       ├── index.ui             # Visual step editor
│       └── save_steps.dspy      # Save DAG configuration
├── models/              # Table definitions (JSON)
├── json/                # CRUD definitions
├── scripts/load_path.py # RBAC registration
└── pyproject.toml

Data Model

Table	Key Fields	Purpose
pipelines	id, name, description	Pipeline definition
pipeline_steps	id, pipeline_id, step_name, step_type, step_order, deps (JSON)	Step DAG nodes
pipeline_versions	id, pipeline_id, version, config (JSON)	Version snapshots
llm	id, name, provider, model, api_key	LLM model config

DSPY Endpoints

Pipeline Management

• pipelines_create.dspy / pipelines_update.dspy / pipelines_delete.dspy — standard CRUD
• get_pipelines.dspy — list with optional filters

Step Orchestration

• pipeline_steps_create.dspy / pipeline_steps_update.dspy / pipeline_steps_delete.dspy
• get_pipeline_steps.dspy — list steps by pipeline_id
• get_step_types.dspy — enumerate available step type handlers
• save_steps.dspy — batch save full DAG from editor UI

Version Control

• pipeline_versions_create.dspy / pipeline_versions_update.dspy / pipeline_versions_delete.dspy
• pipeline_publish.dspy — snapshot + activate version

Pitfalls

• sor.R() 3-arg → sqlExe: Do NOT use sor.R('table', where, sort). Use sor.sqlExe("SELECT ... ORDER BY ...", params).
• get_sor_context not DBPools().sqlorContext: DSPY files must use get_sor_context(env, 'pipeline').
• CRUD generated dirs: wwwroot/pipeline_steps/, wwwroot/pipeline_versions/, etc. are auto-generated by xls2ui. Never edit directly — modify json/*.json and re-run build.sh.
• load_path.py: Every new .dspy must be registered. No wildcards.