yumoqing/sage_datamart
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
57 Commits 1 Branch 0 Tags
Commit Graph

57 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
yumoqing
21ad2403a0 fix: api_provider_roi JOIN organization.orgname + ChartScatter nameField改为provider_name 2026-07-23 11:54:58 +08:00
yumoqing
698dc940b4 fix: ChartPie加pie_options/ChartScatter改xField/yField/sizeField 2026-07-23 11:21:32 +08:00
yumoqing
dbcf33ff8c fix: get_realtime_stats用int(r.xxx or 0)替代hasattr,处理SQL返回NULL 2026-07-23 11:17:55 +08:00
yumoqing
135b394b2a fix: rt_card_amount简化为amt|float,去掉if/else避免jinja2运算歧义 2026-07-23 11:07:43 +08:00
yumoqing
94147b24cc fix: sect_*.ui去entire_url改用绝对路径(子页面entire_url不可用) 2026-07-23 10:45:47 +08:00
yumoqing
41db2df9c9 fix: 使用bricks原生ChartBar/ChartPie/ChartScatter替代WebWidget+Jinja2 2026-07-22 18:54:12 +08:00
yumoqing
a31d0d3bbb fix: 图表改用原生CSS进度条(VBox+bgcolor),去掉不存在的WebWidget 2026-07-22 18:50:46 +08:00
yumoqing
12db192320 fix: api_model_perf/api_provider_roi params_kw=None时.get()→AttributeError,改用(params_kw or {}) 2026-07-22 18:39:55 +08:00
yumoqing
e86ff06b98 fix: organization列名 name→orgname 对照model定义 2026-07-22 18:08:06 +08:00
yumoqing
bdffa5c81b fix: 模型延迟统一用COALESCE(ttft,ttot)覆盖异步模型 2026-07-22 18:01:06 +08:00
yumoqing
85913f8332 fix: 模型性能排行按model归并(GROUP BY),消除同模型多条记录 2026-07-22 17:56:18 +08:00
yumoqing
3b9391b582 fix: WebWidget script用{%set%}+json.dumps包裹,修复JSON转义错误 2026-07-22 17:54:49 +08:00
yumoqing
8da46acbce fix: 供应商排行JOIN organization.name获取orgname 2026-07-22 17:42:23 +08:00
yumoqing
aeb5950cd0 fix: 供应商排行JOIN表名 llmprovider→modelprovider 2026-07-22 17:28:28 +08:00
yumoqing
47278c16ec ui: 实时卡片缩小字体+分组(今天调用/用户)两行展示 2026-07-22 17:24:16 +08:00
yumoqing
dbe45ed489 feat: 6个监控组件图表化(环形图+柱状图+气泡图+美化列表) 2026-07-22 17:18:07 +08:00
yumoqing
ef83842af2 fix: 用户排行JOIN users.username,供应商排行JOIN llmprovider.name 2026-07-22 16:53:01 +08:00
yumoqing
a229abfb81 fix: load_path.py Sage路径 ~/repos/sage→~/sage 适配生产环境 2026-07-22 16:09:35 +08:00
yumoqing
7db361425b feat: 6个监控组件改用RefreshWidget+Jinja2服务端渲染,初始即加载+60s刷新 2026-07-22 14:56:20 +08:00
yumoqing
482c0fac8b fix: _get_realtime_snapshot加请求级缓存,一次页面渲染只查一次DB 2026-07-22 13:39:37 +08:00
yumoqing
305395ca99 fix: 4个DSPY统一用env.api_xxx()调用,消除namespace注入不稳定 2026-07-22 13:33:42 +08:00
yumoqing
ef5644498e feat: 实时卡片统一用get_realtime_stats一次查llmusage,消除数据矛盾 2026-07-22 13:29:02 +08:00
yumoqing
b129fc2ef2 fix: 平台管理员(userorgid=0)不过滤机构,看全部数据 2026-07-22 12:05:58 +08:00
yumoqing
ac40a08021 debug: 所有try/except加exception日志,可追踪API失败的真正原因 2026-07-22 11:49:24 +08:00
yumoqing
1ca9af7a28 fix: 仪表盘查询去掉call_date=today限制,显示全量历史数据 2026-07-22 11:33:59 +08:00
yumoqing
3439ce0a2c fix: dm_model_perf_daily.catelogid 10→32 + GROUP BY userorgid加表前缀 + 独立聚合脚本aggregate_all.py 2026-07-22 10:48:00 +08:00
yumoqing
72eeba1c2a fix: GROUP BY currency加表前缀dm_model_call_fact消除ambiguous警告 2026-07-21 18:30:34 +08:00
yumoqing
6fb1b8cf44 fix: DSPY格式统一为Python脚本+json.dumps,消除JSON格式导致的NoneType异常 2026-07-21 18:16:23 +08:00
yumoqing
0a35be5101 fix: 补全所有j2_*和api_model_perf/api_provider_roi的try/except,彻底消除None返回 2026-07-21 18:15:03 +08:00
yumoqing
e0399c4178 fix: 所有API函数加try/except防止get_sor_context异常返回None 2026-07-21 18:05:11 +08:00
yumoqing
abc457d547 fix: 去掉get_sor_context包装,sor已传入直接用 2026-07-21 17:54:48 +08:00
yumoqing
33447fbbe5 fix: _get_exchange_rates/_load_usage_log_bills env取sor.env,无则ServerEnv()兜底 2026-07-21 17:52:18 +08:00
yumoqing
5030e0640c fix: _get_exchange_rates/_load_usage_log_bills直接用传入sor,去掉get_sor_context(None)导致的NoneType错误 2026-07-21 17:45:12 +08:00
yumoqing
ed0107347e fix: dm_model_call_fact.catelogid长度10→32 (digital_human等超长catelogid) 2026-07-21 17:25:19 +08:00
yumoqing
237c09262f fix: init_data.py补全缺失字段+userorgid为None兜底(空串),支持重复执行 2026-07-21 15:05:02 +08:00
yumoqing
52acdbbfab fix: LIMIT语法错误 - str(limit)导致LIMIT '5',改为int 2026-07-21 14:30:38 +08:00
yumoqing
993c293b56 fix: users表列名修正 userorgid→orgid 2026-07-21 14:06:05 +08:00
yumoqing
05584d8871 feat: 实时数据Dashboard+客户/分销商角色视图 
- dashboards.py: 新增实时指标(直查llmusage,无ETL延迟) + 分销商专用API + scope过滤版
- init.py: 角色检测(_resolve_view),实时卡片J2函数,分销商API端点
- index.ui: 7个实时卡片(15s刷新)替换旧stat_*卡片
- 新增rt_card_*.ui(7个)+api端点(2个)
- load_path.py: 注册新UI路径
 2026-07-21 12:03:28 +08:00
yumoqing
43a115a14d ETL: 金额取product_usage_log.sell_price实际计费金额,币种取purchase_currency,多币种分别统计并并账到CNY 2026-07-20 16:10:17 +08:00
yumoqing
4d8e93dc87 完整修复币种逻辑： 
- dm_model_call_fact 增加 amount_cny 字段（CNY折算金额）
- ETL sync_call_fact 同步时计算并写入 amount_cny
- dm_model_perf_daily/dm_provider_cost_daily 唯一键加入 currency（按币种分组）
- ETL 聚合函数按 currency 分组，保留原币种维度
- Dashboard 查询统一使用 amount_cny 进行跨币种比较
- 汇率查询从 accounting 模块获取（修正库名）
- 新增币种分布统计接口和UI展示
 2026-07-20 15:44:36 +08:00
yumoqing
e84fc8882b refactor: remove DDL from ETL/init, use llm_api_map for catelogid 
- Remove ensure_tables() from etl.py (DDL belongs in models/*.json + build.sh)
- Use llm_api_map.isdefaultcatelog='1' to get catelogid (llm table no longer has catelogid)
- Fix currency field: use amount_currency from llmusage_history
- init_data.py: remove ensure_tables import and call, reduce to 3 steps
 2026-07-20 14:11:06 +08:00
yumoqing
c78f3be577 cleanup: remove obsolete sage_datamart/init_data.py (moved to scripts/init_data.py) 2026-07-20 13:46:03 +08:00
yumoqing
2f9a890128 refactor: move init_data to standalone script (not cron endpoint) 
- scripts/init_data.py: standalone CLI tool, connects directly to sage DB
- Remove HTTP endpoint (cron/init_data.dspy) - init is one-time operation
- Remove init_datamart from init.py - no longer exposed as web service
- Remove RBAC permission for init_data.dspy

Usage: cd /path/to/sage && python /path/to/sage_datamart/scripts/init_data.py
 2026-07-20 13:44:09 +08:00
yumoqing
7f5c1ed372 feat: add init_data script to import historical data from llmusage + llmusage_history into data mart 
- init_data.py: batch import with dedup, aggregate all historical dates
- wwwroot/cron/init_data.dspy: trigger endpoint
- init.py: register init_datamart function with IP whitelist
- load_path.py: add RBAC permission for init_data.dspy
 2026-07-20 13:08:04 +08:00
yumoqing
ac97245aa3 fix: dm_model_call_fact created_at use timestamp instead of datetime with invalid NOW() default 2026-07-20 13:01:40 +08:00
yumoqing
eccd6e8f04 feat: add models/ with 3 data mart table definitions (dm_model_call_fact, dm_model_perf_daily, dm_provider_cost_daily) 
- dm_model_call_fact: LLM调用明细事实表 (23 fields)
- dm_model_perf_daily: 模型性能日汇总表 (25 fields)
- dm_provider_cost_daily: 供应商成本日汇总表 (11 fields)
- build.sh: generate DDL from models/*.json via json2ddl
- .gitignore: exclude generated mysql.ddl.sql
 2026-07-20 12:58:17 +08:00
yumoqing
4f23e07bcd fix: cron dspy移除import + load_path.py添加cron权限(any) + stat_total_users 
- cron/*.dspy: 移除import，改为调用init.py注册的cron_etl_*函数
- init.py: 新增cron_etl_sync/aggregate/provider_cost函数（含IP白名单校验）
- load_path.py: 添加3个cron dspy路径为any角色，补充stat_total_users.ui
- dspy内权限: request['client_ip']不在127.0.0.1则拒绝
 2026-07-20 10:55:05 +08:00
yumoqing
8a97ea4a86 refactor: rename sage_datamark → sage_datamart + add ETL cron + stat UIs 
- Rename package directory and all references from sage_datamark to sage_datamart
- Add 3 ETL cron dspy files (sync/aggregate/provider_cost) for scheduled data pipelines
- Add 5 stat UI files (total_users/fail_calls/success_rate/avg_ttft/total_calls)
- Register j2_ template functions in init.py for UI rendering
- Move j2_avg_ttft SQL to dashboards.py for DRY
- Clean unused imports (getConfig/DBPools/exception)
- Update git remote to sage_datamart.git
 2026-07-20 10:49:15 +08:00
yumoqing
0f3c7f23f4 fix: add import json to model_perf.dspy and provider_roi.dspy 2026-07-17 18:12:56 +08:00
yumoqing
64750630e1 docs: updated API list with dashboard_for_sage replacements 2026-07-17 18:11:49 +08:00