yumoqing/sage_datamart
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
30 Commits 1 Branch 0 Tags
Commit Graph

30 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
yumoqing
6fb1b8cf44 fix: DSPY格式统一为Python脚本+json.dumps,消除JSON格式导致的NoneType异常 2026-07-21 18:16:23 +08:00
yumoqing
0a35be5101 fix: 补全所有j2_*和api_model_perf/api_provider_roi的try/except,彻底消除None返回 2026-07-21 18:15:03 +08:00
yumoqing
e0399c4178 fix: 所有API函数加try/except防止get_sor_context异常返回None 2026-07-21 18:05:11 +08:00
yumoqing
abc457d547 fix: 去掉get_sor_context包装,sor已传入直接用 2026-07-21 17:54:48 +08:00
yumoqing
33447fbbe5 fix: _get_exchange_rates/_load_usage_log_bills env取sor.env,无则ServerEnv()兜底 2026-07-21 17:52:18 +08:00
yumoqing
5030e0640c fix: _get_exchange_rates/_load_usage_log_bills直接用传入sor,去掉get_sor_context(None)导致的NoneType错误 2026-07-21 17:45:12 +08:00
yumoqing
ed0107347e fix: dm_model_call_fact.catelogid长度10→32 (digital_human等超长catelogid) 2026-07-21 17:25:19 +08:00
yumoqing
237c09262f fix: init_data.py补全缺失字段+userorgid为None兜底(空串),支持重复执行 2026-07-21 15:05:02 +08:00
yumoqing
52acdbbfab fix: LIMIT语法错误 - str(limit)导致LIMIT '5',改为int 2026-07-21 14:30:38 +08:00
yumoqing
993c293b56 fix: users表列名修正 userorgid→orgid 2026-07-21 14:06:05 +08:00
yumoqing
05584d8871 feat: 实时数据Dashboard+客户/分销商角色视图 
- dashboards.py: 新增实时指标(直查llmusage,无ETL延迟) + 分销商专用API + scope过滤版
- init.py: 角色检测(_resolve_view),实时卡片J2函数,分销商API端点
- index.ui: 7个实时卡片(15s刷新)替换旧stat_*卡片
- 新增rt_card_*.ui(7个)+api端点(2个)
- load_path.py: 注册新UI路径
 2026-07-21 12:03:28 +08:00
yumoqing
43a115a14d ETL: 金额取product_usage_log.sell_price实际计费金额,币种取purchase_currency,多币种分别统计并并账到CNY 2026-07-20 16:10:17 +08:00
yumoqing
4d8e93dc87 完整修复币种逻辑： 
- dm_model_call_fact 增加 amount_cny 字段（CNY折算金额）
- ETL sync_call_fact 同步时计算并写入 amount_cny
- dm_model_perf_daily/dm_provider_cost_daily 唯一键加入 currency（按币种分组）
- ETL 聚合函数按 currency 分组，保留原币种维度
- Dashboard 查询统一使用 amount_cny 进行跨币种比较
- 汇率查询从 accounting 模块获取（修正库名）
- 新增币种分布统计接口和UI展示
 2026-07-20 15:44:36 +08:00
yumoqing
e84fc8882b refactor: remove DDL from ETL/init, use llm_api_map for catelogid 
- Remove ensure_tables() from etl.py (DDL belongs in models/*.json + build.sh)
- Use llm_api_map.isdefaultcatelog='1' to get catelogid (llm table no longer has catelogid)
- Fix currency field: use amount_currency from llmusage_history
- init_data.py: remove ensure_tables import and call, reduce to 3 steps
 2026-07-20 14:11:06 +08:00
yumoqing
c78f3be577 cleanup: remove obsolete sage_datamart/init_data.py (moved to scripts/init_data.py) 2026-07-20 13:46:03 +08:00
yumoqing
2f9a890128 refactor: move init_data to standalone script (not cron endpoint) 
- scripts/init_data.py: standalone CLI tool, connects directly to sage DB
- Remove HTTP endpoint (cron/init_data.dspy) - init is one-time operation
- Remove init_datamart from init.py - no longer exposed as web service
- Remove RBAC permission for init_data.dspy

Usage: cd /path/to/sage && python /path/to/sage_datamart/scripts/init_data.py
 2026-07-20 13:44:09 +08:00
yumoqing
7f5c1ed372 feat: add init_data script to import historical data from llmusage + llmusage_history into data mart 
- init_data.py: batch import with dedup, aggregate all historical dates
- wwwroot/cron/init_data.dspy: trigger endpoint
- init.py: register init_datamart function with IP whitelist
- load_path.py: add RBAC permission for init_data.dspy
 2026-07-20 13:08:04 +08:00
yumoqing
ac97245aa3 fix: dm_model_call_fact created_at use timestamp instead of datetime with invalid NOW() default 2026-07-20 13:01:40 +08:00
yumoqing
eccd6e8f04 feat: add models/ with 3 data mart table definitions (dm_model_call_fact, dm_model_perf_daily, dm_provider_cost_daily) 
- dm_model_call_fact: LLM调用明细事实表 (23 fields)
- dm_model_perf_daily: 模型性能日汇总表 (25 fields)
- dm_provider_cost_daily: 供应商成本日汇总表 (11 fields)
- build.sh: generate DDL from models/*.json via json2ddl
- .gitignore: exclude generated mysql.ddl.sql
 2026-07-20 12:58:17 +08:00
yumoqing
4f23e07bcd fix: cron dspy移除import + load_path.py添加cron权限(any) + stat_total_users 
- cron/*.dspy: 移除import，改为调用init.py注册的cron_etl_*函数
- init.py: 新增cron_etl_sync/aggregate/provider_cost函数（含IP白名单校验）
- load_path.py: 添加3个cron dspy路径为any角色，补充stat_total_users.ui
- dspy内权限: request['client_ip']不在127.0.0.1则拒绝
 2026-07-20 10:55:05 +08:00
yumoqing
8a97ea4a86 refactor: rename sage_datamark → sage_datamart + add ETL cron + stat UIs 
- Rename package directory and all references from sage_datamark to sage_datamart
- Add 3 ETL cron dspy files (sync/aggregate/provider_cost) for scheduled data pipelines
- Add 5 stat UI files (total_users/fail_calls/success_rate/avg_ttft/total_calls)
- Register j2_ template functions in init.py for UI rendering
- Move j2_avg_ttft SQL to dashboards.py for DRY
- Clean unused imports (getConfig/DBPools/exception)
- Update git remote to sage_datamart.git
 2026-07-20 10:49:15 +08:00
yumoqing
0f3c7f23f4 fix: add import json to model_perf.dspy and provider_roi.dspy 2026-07-17 18:12:56 +08:00
yumoqing
64750630e1 docs: updated API list with dashboard_for_sage replacements 2026-07-17 18:11:49 +08:00
yumoqing
054b0dd8e2 feat: unified dashboard — all dashboard_for_sage metrics from data mart, 6 stat cards, perf+ops 2026-07-17 18:11:19 +08:00
yumoqing
92572bcf6b docs: correct org hierarchy from supplychain, not organization.parentid 2026-07-17 18:06:14 +08:00
yumoqing
565c44d130 docs: README with data sources, frequency, business logic, API docs 2026-07-17 17:23:58 +08:00
yumoqing
32e9dc5cb8 Merge branch 'main' of git.opencomputing.cn:yumoqing/sage_datamark 2026-07-17 17:20:25 +08:00
yumoqing
8aafb31ef4 rename: data_mart → sage_datamark 2026-07-17 17:20:13 +08:00
yumoqing
7b85f024ee Initial commit 2026-07-17 17:20:04 +08:00
yumoqing
4d24094048 feat: data_mart module — model perf monitoring + provider ROI dashboard 2026-07-17 16:56:52 +08:00