64 lines
2.7 KiB
Markdown
64 lines
2.7 KiB
Markdown
---
|
|
name: pipeline-core
|
|
description: "Pipeline core module — pipeline definition, step orchestration, version control, and publishing."
|
|
---
|
|
|
|
# pipeline_core
|
|
|
|
Core module for managing pipeline definitions, step DAG orchestration, version snapshots, and publishing workflow.
|
|
|
|
## Directory Layout
|
|
|
|
```
|
|
pipeline_core/
|
|
├── pipeline_core/ # Python package
|
|
│ ├── __init__.py
|
|
│ └── init.py # ServerEnv registration via load_pipeline_core()
|
|
├── wwwroot/
|
|
│ ├── index.ui # Module entry — pipeline list dashboard
|
|
│ ├── api/
|
|
│ │ ├── pipelines_*.dspy # CRUD wrappers for pipelines table
|
|
│ │ ├── pipeline_steps_*.dspy # CRUD wrappers for pipeline_steps
|
|
│ │ ├── pipeline_versions_*.dspy
|
|
│ │ └── pipeline_publish.dspy # Version publish + activate
|
|
│ └── pipeline_editor/
|
|
│ ├── index.ui # Visual step editor
|
|
│ └── save_steps.dspy # Save DAG configuration
|
|
├── models/ # Table definitions (JSON)
|
|
├── json/ # CRUD definitions
|
|
├── scripts/load_path.py # RBAC registration
|
|
└── pyproject.toml
|
|
```
|
|
|
|
## Data Model
|
|
|
|
| Table | Key Fields | Purpose |
|
|
|-------|-----------|---------|
|
|
| `pipelines` | id, name, description | Pipeline definition |
|
|
| `pipeline_steps` | id, pipeline_id, step_name, step_type, step_order, deps (JSON) | Step DAG nodes |
|
|
| `pipeline_versions` | id, pipeline_id, version, config (JSON) | Version snapshots |
|
|
| `llm` | id, name, provider, model, api_key | LLM model config |
|
|
|
|
## DSPY Endpoints
|
|
|
|
### Pipeline Management
|
|
- `pipelines_create.dspy` / `pipelines_update.dspy` / `pipelines_delete.dspy` — standard CRUD
|
|
- `get_pipelines.dspy` — list with optional filters
|
|
|
|
### Step Orchestration
|
|
- `pipeline_steps_create.dspy` / `pipeline_steps_update.dspy` / `pipeline_steps_delete.dspy`
|
|
- `get_pipeline_steps.dspy` — list steps by pipeline_id
|
|
- `get_step_types.dspy` — enumerate available step type handlers
|
|
- `save_steps.dspy` — batch save full DAG from editor UI
|
|
|
|
### Version Control
|
|
- `pipeline_versions_create.dspy` / `pipeline_versions_update.dspy` / `pipeline_versions_delete.dspy`
|
|
- `pipeline_publish.dspy` — snapshot + activate version
|
|
|
|
## Pitfalls
|
|
|
|
- **sor.R() 3-arg → sqlExe**: Do NOT use `sor.R('table', where, sort)`. Use `sor.sqlExe("SELECT ... ORDER BY ...", params)`.
|
|
- **get_sor_context not DBPools().sqlorContext**: DSPY files must use `get_sor_context(env, 'pipeline')`.
|
|
- **CRUD generated dirs**: `wwwroot/pipeline_steps/`, `wwwroot/pipeline_versions/`, etc. are auto-generated by `xls2ui`. Never edit directly — modify `json/*.json` and re-run `build.sh`.
|
|
- **load_path.py**: Every new .dspy must be registered. No wildcards.
|