---
name: pipeline-core
description: "Pipeline core module — pipeline definition, step orchestration, version control, and publishing."
---

# pipeline_core

Core module for managing pipeline definitions, step DAG orchestration, version snapshots, and publishing workflow.

## Directory Layout

```
pipeline_core/
├── pipeline_core/       # Python package
│   ├── __init__.py
│   └── init.py          # ServerEnv registration via load_pipeline_core()
├── wwwroot/
│   ├── index.ui         # Module entry — pipeline list dashboard
│   ├── api/
│   │   ├── pipelines_*.dspy      # CRUD wrappers for pipelines table
│   │   ├── pipeline_steps_*.dspy # CRUD wrappers for pipeline_steps
│   │   ├── pipeline_versions_*.dspy
│   │   └── pipeline_publish.dspy # Version publish + activate
│   └── pipeline_editor/
│       ├── index.ui             # Visual step editor
│       └── save_steps.dspy      # Save DAG configuration
├── models/              # Table definitions (JSON)
├── json/                # CRUD definitions
├── scripts/load_path.py # RBAC registration
└── pyproject.toml
```

## Data Model

| Table | Key Fields | Purpose |
|-------|-----------|---------|
| `pipelines` | id, name, description | Pipeline definition |
| `pipeline_steps` | id, pipeline_id, step_name, step_type, step_order, deps (JSON) | Step DAG nodes |
| `pipeline_versions` | id, pipeline_id, version, config (JSON) | Version snapshots |
| `llm` | id, name, provider, model, api_key | LLM model config |

## DSPY Endpoints

### Pipeline Management
- `pipelines_create.dspy` / `pipelines_update.dspy` / `pipelines_delete.dspy` — standard CRUD
- `get_pipelines.dspy` — list with optional filters

### Step Orchestration
- `pipeline_steps_create.dspy` / `pipeline_steps_update.dspy` / `pipeline_steps_delete.dspy`
- `get_pipeline_steps.dspy` — list steps by pipeline_id
- `get_step_types.dspy` — enumerate available step type handlers
- `save_steps.dspy` — batch save full DAG from editor UI

### Version Control
- `pipeline_versions_create.dspy` / `pipeline_versions_update.dspy` / `pipeline_versions_delete.dspy`
- `pipeline_publish.dspy` — snapshot + activate version

## Pitfalls

- **sor.R() 3-arg → sqlExe**: Do NOT use `sor.R('table', where, sort)`. Use `sor.sqlExe("SELECT ... ORDER BY ...", params)`.
- **get_sor_context not DBPools().sqlorContext**: DSPY files must use `get_sor_context(env, 'pipeline')`.
- **CRUD generated dirs**: `wwwroot/pipeline_steps/`, `wwwroot/pipeline_versions/`, etc. are auto-generated by `xls2ui`. Never edit directly — modify `json/*.json` and re-run `build.sh`.
- **load_path.py**: Every new .dspy must be registered. No wildcards.