yumoqing/sage_datamart
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor: rename sage_datamark → sage_datamart + add ETL cron + stat UIs

Browse Source 
- Rename package directory and all references from sage_datamark to sage_datamart
- Add 3 ETL cron dspy files (sync/aggregate/provider_cost) for scheduled data pipelines
- Add 5 stat UI files (total_users/fail_calls/success_rate/avg_ttft/total_calls)
- Register j2_ template functions in init.py for UI rendering
- Move j2_avg_ttft SQL to dashboards.py for DRY
- Clean unused imports (getConfig/DBPools/exception)
- Update git remote to sage_datamart.git
This commit is contained in:
yumoqing 2026-07-20 10:49:15 +08:00
parent 0f3c7f23f4
commit 8a97ea4a86
21 changed files with 137 additions and 61 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

32
README.md
View File

@ -62,28 +62,28 @@
## Dashboard 看板
### 统一数据看板 `/sage_datamark/index.ui`
### 统一数据看板 `/sage_datamart/index.ui`
- 6 个统计卡片：今日调用、今日金额、成功率、失败数、平均TTFT、总用户
- 模型性能排行 + 供应商性价比对比（双栏）
- 热门模型 / 用户排行 / 供应商排行（三栏）
- 数据来源全部从集市表 `dm_model_call_fact` / `dm_model_perf_daily` / `dm_provider_cost_daily`
### 供应商性价比看板 `/sage_datamark/provider_roi.ui`
### 供应商性价比看板 `/sage_datamart/provider_roi.ui`
- 同模型不同供应商对比：单价(¥/token)、TTFT、调用次数、总费用
## API 接口
| 端点 | 说明 | 替代 dashboard_for_sage |
|------|------|------------------------|
| `/sage_datamark/api/stats.dspy` | 统计卡片 | get_today_usage/amount, active_users |
| `/sage_datamark/api/model_perf.dspy` | 模型性能 | - (新增) |
| `/sage_datamark/api/provider_roi.dspy` | 供应商性价比 | - (新增) |
| `/sage_datamark/api/top_models.dspy` | 热门模型排行 | get_top_models |
| `/sage_datamark/api/top_providers.dspy` | 供应商排行 | get_top_providers_combined |
| `/sage_datamark/api/top_users.dspy` | 用户排行 | get_top_users_combined |
| `/sage_datamark/api/customer_models.dspy` | 客户今日+当月用量 | get_customer_daily/monthly_models |
| `/sage_datamark/api/daily_trend.dspy` | 7天趋势 | - (新增) |
| `/sage_datamark/api/hourly_concurrency.dspy` | 每小时并发 | - (新增) |
| `/sage_datamart/api/stats.dspy` | 统计卡片 | get_today_usage/amount, active_users |
| `/sage_datamart/api/model_perf.dspy` | 模型性能 | - (新增) |
| `/sage_datamart/api/provider_roi.dspy` | 供应商性价比 | - (新增) |
| `/sage_datamart/api/top_models.dspy` | 热门模型排行 | get_top_models |
| `/sage_datamart/api/top_providers.dspy` | 供应商排行 | get_top_providers_combined |
| `/sage_datamart/api/top_users.dspy` | 用户排行 | get_top_users_combined |
| `/sage_datamart/api/customer_models.dspy` | 客户今日+当月用量 | get_customer_daily/monthly_models |
| `/sage_datamart/api/daily_trend.dspy` | 7天趋势 | - (新增) |
| `/sage_datamart/api/hourly_concurrency.dspy` | 每小时并发 | - (新增) |
### model_perf.dspy
```json
@ -124,25 +124,25 @@
```bash
cd ~/sage/pkgs
git clone git@git.opencomputing.cn:yumoqing/sage_datamark.git
cd sage_datamark && pip install .
git clone git@git.opencomputing.cn:yumoqing/sage_datamart.git
cd sage_datamart && pip install .
py3/bin/python scripts/load_path.py
# 添加 ETL cron 任务（在 Sage start.sh 或系统 cron）
# 每 5 分钟同步
*/5 * * * * cd ~/sage && py3/bin/python -c "
from sage_datamark.etl import run_etl_sync
from sage_datamart.etl import run_etl_sync
from sqlor.dbpools import DBPools
from appPublic.jsonConfig import getConfig
import asyncio
config = getConfig('.')
db = DBPools(config.databases)
async def go():
env_dbname = ServerEnv().get_module_dbname('sage_datamark')
env_dbname = ServerEnv().get_module_dbname('sage_datamart')
async with db.sqlorContext(env_dbname) as sor:
await run_etl_sync(sor)
asyncio.run(go())
"
# 重启 Sage，访问 /sage_datamark/index.ui
# 重启 Sage，访问 /sage_datamart/index.ui
```

2
build.sh
View File

@ -1,4 +1,4 @@
#!/bin/bash
set -e
cd "$(dirname "$0")"
echo "[sage_datamark] build done"
echo "[sage_datamart] build done"

4
json/build.sh
View File

@ -1,3 +1,3 @@
#!/bin/bash
# sage_datamark has no CRUD tables - skip xls2ui
echo "[sage_datamark] json build skipped (no CRUD)"
# sage_datamart has no CRUD tables - skip xls2ui
echo "[sage_datamart] json build skipped (no CRUD)"

4
pyproject.toml
View File

@ -3,10 +3,10 @@ requires = ["setuptools>=61", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "sage_datamark"
name = "sage_datamart"
version = "0.1.0"
description = "Sage 数据集市 — 模型性能指标、供应商性价比、多维度分析"
requires-python = ">=3.10"
[tool.setuptools.packages.find]
include = ["sage_datamark*"]
include = ["sage_datamart*"]

2
sage_datamark/__init__.py
View File

@ -1,2 +0,0 @@
"""sage_datamark — Sage 数据集市模块"""
from .init import load_sage_datamark

2
sage_datamart/__init__.py Normal file
View File

@ -0,0 +1,2 @@
"""sage_datamart — Sage 数据集市模块"""
from .init import load_sage_datamart

9
sage_datamark/dashboards.py → sage_datamart/dashboards.py
View File

@ -48,6 +48,15 @@ async def get_active_users(env, sor):
return int(recs[0].cnt) if recs else 0
async def get_avg_ttft(env, sor):
"""当天平均 TTFT"""
sql = "SELECT AVG(avg_ttft_ms) as v FROM dm_model_perf_daily WHERE stat_date = ${today}$"
recs = await sor.sqlExe(sql, {'today': env.curDateString()})
if recs and recs[0].v:
return round(float(recs[0].v), 0)
return 0
# ── 排行榜 ──
async def get_top_models(env, sor, limit=5):

8
sage_datamark/etl.py → sage_datamart/etl.py
View File

@ -11,7 +11,7 @@ from appPublic.log import debug, info, exception
from sqlor.dbpools import DBPools
from ahserver.serverenv import ServerEnv
MODULE_NAME = "sage_datamark"
MODULE_NAME = "sage_datamart"
async def ensure_tables(sor):
@ -129,7 +129,7 @@ async def sync_call_fact(sor, last_sync=None):
ORDER BY lu.use_time
"""
rows = await sor.sqlExe(sql, {'last_sync': last_sync})
info('sage_datamark ETL: syncing ' + str(len(rows)) + ' rows from ' + last_sync)
info('sage_datamart ETL: syncing ' + str(len(rows)) + ' rows from ' + last_sync)
count = 0
for r in rows:
@ -168,7 +168,7 @@ async def sync_call_fact(sor, last_sync=None):
await sor.C('dm_model_call_fact', ns)
count += 1
info('sage_datamark ETL: inserted ' + str(count) + ' new rows')
info('sage_datamart ETL: inserted ' + str(count) + ' new rows')
return count
@ -195,7 +195,7 @@ async def aggregate_daily_perf(sor, stat_date=None):
GROUP BY call_date, llmid, model, catelogid, providerid, ownerid, userorgid
"""
rows = await sor.sqlExe(sql, {'stat_date': stat_date})
info('sage_datamark AGG: aggregating ' + str(len(rows)) + ' groups for ' + stat_date)
info('sage_datamart AGG: aggregating ' + str(len(rows)) + ' groups for ' + stat_date)
for r in rows:
r = DictObject(r)

56
sage_datamark/init.py → sage_datamart/init.py
View File

@ -1,17 +1,17 @@
"""sage_datamark 模块初始化 — 统一 Dashboard API"""
"""sage_datamart 模块初始化 — 统一 Dashboard API"""
from ahserver.serverenv import ServerEnv
from appPublic.jsonConfig import getConfig
from appPublic.log import debug, exception
from sqlor.dbpools import DBPools, get_sor_context
from appPublic.log import debug
from sqlor.dbpools import get_sor_context
from .etl import sync_call_fact, aggregate_daily_perf, aggregate_provider_cost, ensure_tables
from .dashboards import (
get_today_usage, get_today_amount, get_success_rate, get_fail_count,
get_active_users, get_top_models, get_top_providers, get_top_users,
get_customer_daily_models, get_customer_monthly_models,
get_customer_user_today, get_daily_trend, get_hourly_concurrency,
get_avg_ttft,
)
MODULE_NAME = "sage_datamark"
MODULE_NAME = "sage_datamart"
# ── 性能指标 API ──
@ -51,7 +51,7 @@ async def api_provider_roi(sor, params_kw=None):
# ── 运营指标 API（替代 dashboard_for_sage）──
async def api_stats(request):
"""统计卡片数据 — /sage_datamark/api/stats.dspy"""
"""统计卡片数据 — /sage_datamart/api/stats.dspy"""
env = request._run_ns
async with get_sor_context(env, MODULE_NAME) as sor:
return {
@ -64,28 +64,28 @@ async def api_stats(request):
async def api_top_models(request):
"""Top 模型 — /sage_datamark/api/top_models.dspy"""
"""Top 模型 — /sage_datamart/api/top_models.dspy"""
env = request._run_ns
async with get_sor_context(env, MODULE_NAME) as sor:
return await get_top_models(env, sor)
async def api_top_providers(request):
"""Top 供应商 — /sage_datamark/api/top_providers.dspy"""
"""Top 供应商 — /sage_datamart/api/top_providers.dspy"""
env = request._run_ns
async with get_sor_context(env, MODULE_NAME) as sor:
return await get_top_providers(env, sor)
async def api_top_users(request):
"""Top 用户 — /sage_datamark/api/top_users.dspy"""
"""Top 用户 — /sage_datamart/api/top_users.dspy"""
env = request._run_ns
async with get_sor_context(env, MODULE_NAME) as sor:
return await get_top_users(env, sor)
async def api_customer_models(request):
"""客户用量 — /sage_datamark/api/customer_models.dspy"""
"""客户用量 — /sage_datamart/api/customer_models.dspy"""
env = request._run_ns
userorgid = await env.get_userorgid()
async with get_sor_context(env, MODULE_NAME) as sor:
@ -96,14 +96,14 @@ async def api_customer_models(request):
async def api_daily_trend(request):
"""7天趋势 — /sage_datamark/api/daily_trend.dspy"""
"""7天趋势 — /sage_datamart/api/daily_trend.dspy"""
env = request._run_ns
async with get_sor_context(env, MODULE_NAME) as sor:
return await get_daily_trend(env, sor)
async def api_hourly_concurrency(request):
"""每小时并发 — /sage_datamark/api/hourly_concurrency.dspy"""
"""每小时并发 — /sage_datamart/api/hourly_concurrency.dspy"""
env = request._run_ns
async with get_sor_context(env, MODULE_NAME) as sor:
return await get_hourly_concurrency(env, sor)
@ -123,7 +123,31 @@ async def j2_today_amount(request):
return await get_today_amount(env, sor)
def load_sage_datamark():
async def j2_active_users(request):
env = request._run_ns
async with get_sor_context(env, MODULE_NAME) as sor:
return await get_active_users(env, sor)
async def j2_fail_count(request):
env = request._run_ns
async with get_sor_context(env, MODULE_NAME) as sor:
return await get_fail_count(env, sor)
async def j2_success_rate(request):
env = request._run_ns
async with get_sor_context(env, MODULE_NAME) as sor:
return await get_success_rate(env, sor)
async def j2_avg_ttft(request):
env = request._run_ns
async with get_sor_context(env, MODULE_NAME) as sor:
return await get_avg_ttft(env, sor)
def load_sage_datamart():
env = ServerEnv()
env.api_model_perf = api_model_perf
env.api_provider_roi = api_provider_roi
@ -136,4 +160,8 @@ def load_sage_datamark():
env.api_hourly_concurrency = api_hourly_concurrency
env.j2_today_usage = j2_today_usage
env.j2_today_amount = j2_today_amount
debug("[sage_datamark] registered all API endpoints")
env.j2_active_users = j2_active_users
env.j2_fail_count = j2_fail_count
env.j2_success_rate = j2_success_rate
env.j2_avg_ttft = j2_avg_ttft
debug("[sage_datamart] registered all API endpoints")

4
scripts/load_path.py
View File

@ -1,8 +1,8 @@
#!/usr/bin/env python3
"""sage_datamark 模块 load_path"""
"""sage_datamart 模块 load_path"""
import os, sys, subprocess
MOD = "sage_datamark"
MOD = "sage_datamart"
SAGE_ROOT = os.environ.get("SAGE_ROOT", os.path.expanduser("~/sage"))
PYTHON = os.path.join(SAGE_ROOT, "py3", "bin", "python")
SET_PERM = os.path.join(SAGE_ROOT, "set_role_perm.py")

2
wwwroot/api/model_perf.dspy
View File

@ -1,6 +1,6 @@
# 模型性能统计 API
import json
env = request._run_ns
async with get_sor_context(env, 'sage_datamark') as sor:
async with get_sor_context(env, 'sage_datamart') as sor:
rows = await env.api_model_perf(sor, params_kw)
return json.dumps(rows, ensure_ascii=False)

2
wwwroot/api/provider_roi.dspy
View File

@ -1,6 +1,6 @@
# 供应商性价比 API
import json
env = request._run_ns
async with get_sor_context(env, 'sage_datamark') as sor:
async with get_sor_context(env, 'sage_datamart') as sor:
rows = await env.api_provider_roi(sor, params_kw)
return json.dumps(rows, ensure_ascii=False)

12
wwwroot/cron/etl_aggregate.dspy Normal file
View File

@ -0,0 +1,12 @@
# ETL: 每小时聚合 dm_model_call_fact → dm_model_perf_daily
ip = request['client_ip']
if ip not in ['127.0.0.1']:
return ip
import json
from sage_datamart.etl import ensure_tables, aggregate_daily_perf
env = request._run_ns
async with get_sor_context(env, 'sage_datamart') as sor:
await ensure_tables(sor)
count = await aggregate_daily_perf(sor)
return json.dumps({'status': 'ok', 'aggregated': count})

12
wwwroot/cron/etl_provider_cost.dspy Normal file
View File

@ -0,0 +1,12 @@
# ETL: 每天聚合供应商性价比 dm_provider_cost_daily
ip = request['client_ip']
if ip not in ['127.0.0.1']:
return ip
import json
from sage_datamart.etl import ensure_tables, aggregate_provider_cost
env = request._run_ns
async with get_sor_context(env, 'sage_datamart') as sor:
await ensure_tables(sor)
count = await aggregate_provider_cost(sor)
return json.dumps({'status': 'ok', 'aggregated': count})

12
wwwroot/cron/etl_sync.dspy Normal file
View File

@ -0,0 +1,12 @@
# ETL: 每5分钟增量同步 llmusage → dm_model_call_fact
ip = request['client_ip']
if ip not in ['127.0.0.1']:
return ip
import json
from sage_datamart.etl import ensure_tables, sync_call_fact
env = request._run_ns
async with get_sor_context(env, 'sage_datamart') as sor:
await ensure_tables(sor)
count = await sync_call_fact(sor)
return json.dumps({'status': 'ok', 'synced': count})

8
wwwroot/menu.ui
View File

@ -6,16 +6,16 @@
"widgettype": "MenuItem",
"options": {
"label": "模型性能",
"url": "{{entire_url('/sage_datamark/index.ui')}}",
"icon": "{{entire_url('/sage_datamark/imgs/perf.svg')}}"
"url": "{{entire_url('/sage_datamart/index.ui')}}",
"icon": "{{entire_url('/sage_datamart/imgs/perf.svg')}}"
}
},
{
"widgettype": "MenuItem",
"options": {
"label": "供应商性价比",
"url": "{{entire_url('/sage_datamark/provider_roi.ui')}}",
"icon": "{{entire_url('/sage_datamark/imgs/roi.svg')}}"
"url": "{{entire_url('/sage_datamart/provider_roi.ui')}}",
"icon": "{{entire_url('/sage_datamart/imgs/roi.svg')}}"
}
}
]

5
wwwroot/stat_avg_ttft.ui
View File

@ -1,7 +1,4 @@
{% set rows = api_model_perf(request) %}
{% set total = [] %}{% set avg = 0 %}{% set cnt = 0 %}
{% for r in rows %}{% if r.avg_ttft_ms %}{% set avg = avg + r.avg_ttft_ms %}{% set cnt = cnt + 1 %}{% endif %}{% endfor %}
{% if cnt > 0 %}{% set avg = (avg / cnt)|round(0) %}{% endif %}
{% set avg = j2_avg_ttft(request) %}
{
"widgettype": "VBox",
"options": {"css": "card", "padding": "16px", "borderRadius": "12px", "width": "100%"},

4
wwwroot/stat_fail_calls.ui
View File

@ -1,2 +1,2 @@
{% set rows = api_model_perf(request) %}{% set t = 0 %}{% for r in rows %}{% set t = t + r.fail_calls %}{% endfor %}
{"widgettype":"VBox","options":{"css":"card","padding":"16px","borderRadius":"12px","width":"100%"},"subwidgets":[{"widgettype":"Text","options":{"text":"失败数","fontSize":"12px","color":"var(--sage-text-secondary)"}},{"widgettype":"Title2","options":{"text":"{{t}}","fontWeight":"700","marginTop":"4px"}}]}
{% set cnt = j2_fail_count(request) %}
{"widgettype":"VBox","options":{"css":"card","padding":"16px","borderRadius":"12px","width":"100%"},"subwidgets":[{"widgettype":"Text","options":{"text":"失败数","fontSize":"12px","color":"var(--sage-text-secondary)"}},{"widgettype":"Title2","options":{"text":"{{cnt}}","fontWeight":"700","marginTop":"4px"}}]}

5
wwwroot/stat_success_rate.ui
View File

@ -1,5 +1,2 @@
{% set rows = api_model_perf(None) %}
{% set total = 0 %}{% set succ = 0 %}
{% for r in rows %}{% set total = total + r.total_calls %}{% set succ = succ + r.success_calls %}{% endfor %}
{% set rate = (succ * 100 / total)|round(1) if total > 0 else 0 %}
{% set rate = j2_success_rate(request) %}
{"widgettype":"VBox","options":{"css":"card","padding":"16px","borderRadius":"12px","width":"100%"},"subwidgets":[{"widgettype":"Text","options":{"text":"成功率","fontSize":"12px","color":"var(--sage-text-secondary)"}},{"widgettype":"Title2","options":{"text":"{{rate}}%","fontWeight":"700","marginTop":"4px"}}]}

4
wwwroot/stat_total_calls.ui
View File

@ -1,2 +1,2 @@
{% set rows = api_model_perf(request) %}{% set t = 0 %}{% for r in rows %}{% set t = t + r.total_calls %}{% endfor %}
{"widgettype":"VBox","options":{"css":"card","padding":"16px","borderRadius":"12px","width":"100%"},"subwidgets":[{"widgettype":"Text","options":{"text":"今日调用","fontSize":"12px","color":"var(--sage-text-secondary)"}},{"widgettype":"Title2","options":{"text":"{{t}}","fontWeight":"700","marginTop":"4px"}}]}
{% set cnt = j2_today_usage(request) %}
{"widgettype":"VBox","options":{"css":"card","padding":"16px","borderRadius":"12px","width":"100%"},"subwidgets":[{"widgettype":"Text","options":{"text":"今日调用","fontSize":"12px","color":"var(--sage-text-secondary)"}},{"widgettype":"Title2","options":{"text":"{{cnt}}","fontWeight":"700","marginTop":"4px"}}]}

9
wwwroot/stat_total_users.ui Normal file
View File

@ -0,0 +1,9 @@
{% set cnt = j2_active_users(request) %}
{
"widgettype": "VBox",
"options": {"css": "card", "padding": "16px", "borderRadius": "12px", "width": "100%"},
"subwidgets": [
{"widgettype": "Text", "options": {"text": "活跃用户", "fontSize": "12px", "color": "var(--sage-text-secondary)"}},
{"widgettype": "Title2", "options": {"text": "{{cnt}}", "fontWeight": "700", "marginTop": "4px"}}
]
}