yumoqing/pipeline_core
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
pipeline_core/skills_library/all/pricing-module/SKILL.md

50 KiB
Raw Blame History

name description author tags
pricing-module Development patterns, pricing engine logic, and API reference for the pricing module — Sage platform's YAML-driven pricing calculation engine Hermes Agent
pricing
billing
yaml
excel
rules-engine
sage

Pricing Module Skill

Overview

The pricing module is Sage's configurable pricing engine that calculates costs based on YAML-defined pricing rules. It supports multi-dimensional condition matching (between, in, =, >, <, >=, <=), formula evaluation, data mappings, time-based pricing periods, and bulk Excel import/export.

Module Location

~/repos/pricing/

Architecture

pricing_program (定价项目)
    │
    └── pricing_program_timing (定价项目时序) — 1:N, time-scoped, 拉链表
            │
            └── pricing_data (YAML) — {pricing_type, fields: {...}, pricings: [...]}

Timing chain (拉链表): pricing_program_timing uses a chain/zipper pattern — each record has enabled_date → expired_date. When adding a new record via add_pricing_program_timing():

  • New record's expired_date is set to 9999-12-31
  • Previous latest record's expired_date is updated to the new enabled_date
  • This creates a continuous timeline: [old_start, new_start) → [new_start, 9999-12-31)

In Sage: llmage → pricing Flow

llm.ppid ──→ pricing_program.id
                │
                └── buffered_charging(ppid, usages)
                        │
                        ├── get_ppid_pricing(ppid) — fetch pricing_data (cached)
                        ├── get_pricing_from_ymalstr(usages, pricing_data) — match rules
                        └── return [amount, cost]  (no discount applied)

Key Tables

Table Purpose Key Fields
pricing_program Pricing project definition id, name, ownerid, providerid, pricing_belong
pricing_program_timing Time-scoped pricing periods (拉链表) id, ppid, name, enabled_date, expired_date, pricing_data(YAML)
pricing_item Individual pricing line items id, pptid, name, formula, spec_values

ServerEnv Functions (via load_pricing())

Available in .dspy files via globals():

Function Purpose
buffered_charging(ppid, data) Calculate pricing with memory cache (primary method)
pricing_program_charging(sor, ppid, data) Calculate pricing without cache (with sor cursor)
get_pricing_program(ppid) Get pricing program definition
load_pricing_data(pptid, webpath_xlsx) Load pricing data from Excel file
write_pricing_patten(request, ppid) Generate Excel pricing template
test_pricing(pptid, data) Test pricing calculation, return total amount
generate_formula_from_factors(price_factors) Auto-generate formula string from price_factors array
add_pricing_program_timing(env, sor, ns) NEW: Timing chain logic — sets expired_date, updates previous record
get_pricing_display(ppid, model=None) Return structured human-readable pricing data for frontend display. model parameter filters items to a specific model name (exact match + model_mappings). Fallback: if filtering yields empty results, returns all items (for single-model ppids without model filter keys).

Pricing Types

Three pricing modes controlled by pricing_type in pricing_data (default: per_use):

per_use — 按量计费（现有模式）

Standard pay-per-use. Each pricings entry has filters + price_factors + formula.

monthly_bundle — 包月套餐

Contains multiple products with quotas and overage pricing:

pricing_type: monthly_bundle
pricings:
  - name: 企业AI套餐
    plan_level: enterprise                # filter
    base_price:
      amount: 9999
      unit_label: 元/月
    products:
      - product: gpt-4
        label: GPT-4
        included: 1000000                 # monthly quota
        unit_label: Token/月
        overage:                          # overage has its own price_factors
          price_factors:
            - factor: tokens
              label: Token
              unit_price: 0.03
              unit_label: 元/千Token
          formula: 0.03 * float(tokens) / 1000.0

Usage is tracked by charge/accounting module. Each product use calls calculate_bundle_usage() which checks quota, calculates overage cost.

monthly_credits — 包月额度

Buy a credit pool, products consume from it:

pricing_type: monthly_credits
pricings:
  - name: AI消耗包
    credit_amount: 10000
    unit_label: 元
    products:
      - product: gpt-4
        label: GPT-4
        cost_per_unit: 0.03
        unit_label: 元/千Token

Pricing Spec Format — REMOVED (2026-07)

The pricing_spec column on pricing_program has been removed. Field definitions now come exclusively from pricing_data.fields in the YAML, which is already self-describing. The pricing_spec was a redundant copy of the field definitions from pricing_data — removing it simplifies the data model and eliminates sync issues between the two sources.

model:
  type: str
  label: "模型"
  role: filter                            # NEW: filter vs factor
  options:
    - "viduq3-pro"
    - "viduq3-turbo"

resolution:
  type: str
  label: "分辨率"
  role: filter
  options:
    - "1024p"
    - "720p"

duration:
  type: int
  label: "时长"
  role: filter

price:
  type: float
  label: 单价
  role: factor                            # calculation factor, not a filter

Required: price field with type: float (for per_use pricing).

Field properties:

  • type: str, int, float, bool, factor
  • label: Display name
  • role: filter (default, used for matching/display) | factor (calculation only, hidden from filters)
  • options: Dropdown options (also used for matching)
  • value_mode: Match mode (=, between, in, >, >=, <, <=)

Pricing Data Format (pricing_program_timing.pricing_data) — NEW DESIGN

The pricing data uses a declarative per-factor structure. Each pricing item is a single factor with its unit price and unit. Complex formulas are decomposed into independent pricing items.

# 计价单位数值映射（只包含本条记录实际用到的单位）
unit_values:
  百万: 1000000
  秒: 1

# 字段定义（必须包含 price_factors, unit_prices, unit, min_amount）
fields:
  price_factors:
    type: string
    role: factor
    label: "计价因子"
  unit_prices:
    type: float
    role: factor
    label: "单位定价"
  unit:
    type: string
    role: factor
    label: "计价单位"
  min_amount:
    type: float
    role: factor
    label: "最低消费"
    default: 0
  
  # derived 字段：从原始 usage 数据计算衍生字段
  uncached_prompt_tokens:
    type: int
    role: factor
    label: "非缓存输入Token"
    derived: "prompt_tokens - prompt_tokens_details.cached_tokens"
  
  cached_tokens:
    type: int
    role: factor
    label: "缓存Token"
    derived: "prompt_tokens_details.cached_tokens"
  
  # filter fields with range conditions need value_mode: between
  prompt_tokens:
    type: int
    role: filter
    label: "输入Token数"
    value_mode: between
  completion_tokens:
    type: int
    role: factor
    label: "输出Token数"
    value_mode: between
  model:
    type: string
    role: filter
    label: "模型"

# 定价规则 — 每条一个 price_factors，无 formula
pricings:
  # 简单按量定价
  - price_factors: prompt_tokens
    unit_prices: 3.2
    unit: 百万
  - price_factors: completion_tokens
    unit_prices: 16.0
    unit: 百万

  # 区间定价：使用 value_mode: between + 顶层范围条件（NOT filters 数组）
  - price_factors: uncached_prompt_tokens
    unit_prices: 6.0
    unit: 百万
    model: qwen3.7-max
    prompt_tokens: "0 ~= 10000000"  # AND condition, not OR
  
  - price_factors: uncached_prompt_tokens
    unit_prices: 12.0
    unit: 百万
    model: qwen3.7-max
    prompt_tokens: "10000000 ~"  # 超过10M的更高价格

  # 规格定价（按分辨率等 filter 匹配，使用 filters 数组）
  - price_factors: duration
    unit: 秒
    filters:
      - SR: 720
        unit_prices: 0.9
      - SR: 1080
        unit_prices: 1.6
  ### unit_values: Per-record unit mapping (NEW DESIGN)

  The `unit_values` mapping in pricing_data should **only contain units actually used in that record's pricings**, not a global map of all possible units. This keeps each pricing_data self-contained and avoids confusion.

  **Example:**
  ```yaml
  # Record that only uses "秒" (seconds)
  unit_values:
    秒: 1
  fields:
    price_factors: {type: string, role: factor, label: "计价因子"}
    unit_prices: {type: float, role: factor, label: "单位定价"}
    unit: {type: string, role: factor, label: "计价单位"}
  pricings:
    - price_factors: duration
      unit_prices: 0.9
      unit: 秒

Common units in production data:

  • 元/百万tokens → 1000000 (LLM token pricing)
  • 秒 → 1 (video/audio duration)
  • 次 → 1 (per-use flat pricing)
  • 张 → 1 (image generation)

Conversion script pattern: When migrating old data, scan all pricings items to collect unique unit values, then build a minimal unit_values map:

used_units = set()
for p in new_pricings:
    u = p.get('unit', '')
    if u:
        used_units.add(u)

# Map known units to their values
UNIT_VALUES = {
    '元/百万tokens': 1000000,
    '秒': 1,
    '次': 1,
    '张': 1,
    # ... other known units
}

filtered_unit_values = {k: v for k, v in UNIT_VALUES.items() if k in used_units}

Calculation Logic (pseudo-code)

total = 0
for item in pricings:
    factor_name = item['price_factors']
    if factor_name == 'flat':
        total += item['unit_prices']
        continue

    usage_value = usages.get(factor_name, 0)
    unit_price = item.get('unit_prices')

    # Range-based pricing: top-level field with value_mode: between
    # e.g., prompt_tokens: "0 ~= 10000000" — matched via check_value()
    # These are AND conditions with other top-level fields
  
    # Spec-based pricing: filters array with OR logic
    if 'filters' in item:
        for f in item['filters']:
            if match_condition(f, usages):
                unit_price = f['unit_prices']
                break

    unit_val = unit_values.get(item['unit'], 1)
    amount = unit_price * usage_value / unit_val

    # Apply min_amount if defined
    if 'min_amount' in item:
        amount = max(amount, item['min_amount'])

    total += amount

Derived Fields (NEW — 2026-06)

Modern LLM APIs return nested token breakdowns (e.g., prompt_tokens_details.cached_tokens). The pricing engine supports derived fields that compute values from nested usage data:

fields:
  uncached_prompt_tokens:
    type: int
    role: factor
    label: "非缓存输入Token"
    derived: "prompt_tokens - prompt_tokens_details.cached_tokens"

  cached_tokens:
    type: int
    role: factor
    label: "缓存Token"
    derived: "prompt_tokens_details.cached_tokens"

Implementation:

  • Engine evaluates derived expressions before processing pricings
  • Dot notation (prompt_tokens_details.cached_tokens) is converted to underscore (prompt_tokens_details_cached_tokens) for Python eval
  • Nested dict values are flattened: config_data['prompt_tokens_details']['cached_tokens'] → eval_env['prompt_tokens_details_cached_tokens']
  • If derived expression fails, field defaults to 0

Use case: When usage data has {"prompt_tokens": 1075, "prompt_tokens_details": {"cached_tokens": 500}}, the derived field uncached_prompt_tokens computes 1075 - 500 = 575, which can then be priced separately from cached_tokens.

See references/pricing-derived-fields.md for implementation details and testing patterns.

Range Filters vs Spec Filters

Two distinct patterns for conditional pricing:

1. Range Filters (value_mode: between)

For token/duration ranges, use top-level field attributes with value_mode: between:

fields:
  prompt_tokens:
    type: int
    role: filter
    label: "输入Token"
    value_mode: between

pricings:
  - price_factors: uncached_prompt_tokens
    unit_prices: 6.0
    unit: 百万
    prompt_tokens: "0 ~= 10000000"  # AND condition, matched via check_value()

Key: Each pricing item with a range condition is a separate entry. The engine matches ALL top-level fields (AND logic).

2. Spec Filters (filters array)

For categorical matching (resolution, model variants), use filters array with OR logic:

pricings:
  - price_factors: duration
    unit: 秒
    filters:
      - SR: 720
        unit_prices: 0.9
      - SR: 1080
        unit_prices: 1.6

Key: The filters array items are mutually exclusive — only one matches per request (OR logic).

Coverage Analysis

Scenario Supported Example
Simple per-unit (token/duration) ✅ unit_prices * usage / unit_value
Range pricing (token brackets) ✅ filters by prompt_tokens range
Quality/spec pricing (resolution) ✅ filters by SR
Cache/non-cache separate pricing ✅ separate pricing items
Fixed fee (per-use flat) ✅ price_factors: flat
Minimum charge ✅ min_amount field
Cumulative tiered (first 10K free) ❌ current is "match bracket" not "cumulative"

Pricing Matching Rules

get_pricing_from_ymalstr(config_data, yamlstr) matches input data against pricing rules:

  1. Iterates all pricings entries
  2. For each rule, checks every field against input via check_value():
    • = (default): exact match data_value == spec_value
    • between: range match (0 ~ 100 means 0 <= value < 100)
    • in: enumeration match (gpt-4 gpt-3.5)
    • >/</>=/<=`: comparison
  3. All fields must match for a rule to be selected
  4. Executes formula via eval() to calculate amount
  5. Returns all matching rules (not just the first one)

Data Mappings

model_mappings:
  "doubao-seed-2-0-pro-260215": "doubao-seed-2-0-pro"

Input values are mapped through *_mappings before matching.

References

  • references/pricing-test-guide.md — Step-by-step pricing test checklist
  • references/qwen-image-pricing.md — Qwen image model pricing patterns
  • references/uapiio-bricks-format.md — uapiio input_fields bricks 兼容格式及 data/response 字段规范
  • references/rbac-redis-cache.md — rbac userperm.py Redis L2 缓存实现要点与调试经验
  • references/pricing-type-design.md — Design decisions for pricing_type expansion (price_factors, monthly_bundle, monthly_credits, module boundaries with supplychain)
  • references/pricing-display-implementation.md — Display API implementation details, mock-based testing pattern, deployment checklist
  • references/pricing-data-formula-parsing.md — Formula pattern taxonomy, reverse-parsing approach, production data quality pitfalls, two coexisting formats (YAML dict vs JSON array)
  • templates/model-pricing.sql — INSERT template for model pricing with cached/uncached token derived fields
  • references/pricing-data-migration.md — Old-to-new conversion rules, filter propagation pitfalls, deployment steps
  • references/production-pricing-verification.md — End-to-end verification workflow: pull llmusage via bugfix SQL API, join for ppid, recalculate amounts, compare with production billing
  • scripts/convert_all_pricing.py — Reusable conversion script: handles JSON arrays, YAML dicts, flat pricing, range filter propagation

Planned Enhancements (Partially Implemented)

✅ price_factors: Declarative pricing (IMPLEMENTED — new design)

The pricing data format has been redesigned to use declarative per-factor pricing instead of formula-based evaluation. See "Pricing Data Format" section above for the complete new structure.

Key change from original plan: Instead of price_factors being an array of dicts alongside formula, each pricing item IS a single factor with unit_prices and unit. No formula field needed — the engine calculates unit_prices * usage / unit_value.

pricing_type: Three pricing modes

pricing_type: per_use | monthly_bundle | monthly_credits
  • per_use (default, current mode): match conditions → calculate from price_factors + unit_prices
  • monthly_bundle: subscription with product quotas + overage pricing
  • monthly_credits: buy credit balance, products consume from it

role field in fields definition

fields:
  model:
    type: str
    label: 模型
    role: filter     # filter = matching condition shown in "适用条件"
                     # factor = calculation factor (not shown as filter)
                     # default = filter (backward compat)

monthly_bundle structure

pricings:
  - name: 企业AI套餐
    base_price: {amount: 9999, unit_label: 元/月}
    products:
      - product: gpt-4
        label: GPT-4
        included: 1000000          # quota
        unit_label: Token/月
        overage:                   # overage has full price_factors
          price_factors:
            - factor: tokens
              label: Token
              unit_price: 0.03
              unit_label: 元/千Token
          formula: 0.03 * float(tokens) / 1000.0

monthly_credits structure

pricings:
  - name: AI消耗包
    credit_amount: 10000
    unit_label: 元
    products:
      - product: gpt-4
        label: GPT-4
        cost_per_unit: 0.03
        unit_label: 元/千Token

Module separation: pricing vs charging

  • pricing module: defines prices + calculates amounts (stateless, no consumption data)
  • charging module: records consumption, tracks quotas, creates bills
    • subscription (主表): user_id, ppid, pricing_type, base_amount, dates, status
    • subscription_product (明细): sub_id, product, label, included, overage_config(JSON)
    • monthly_usage: sub_id, product, month, total_used, remaining, total_cost
    • usage_record: sub_id, product, usage_data, included_used, overage_used, amount

Multi-level distribution belongs to supplychain module

Distribution layers (supplier → platform → level-1 → level-2) and markup/discount chains are NOT in pricing. Pricing only defines the base price (supplier list price). Supplychain module handles:

  • Distributor hierarchy (parent_id tree)
  • Per-level adjustment (discount/markup/fixed)
  • Per-distributor price calculation (chain multiplication from base)

Backward compatibility

  • No pricing_type → default per_use
  • No role → default filter
  • No price_factors → display API falls back to showing formula
  • New and old data coexist; migrate incrementally

CRUD JSON Provider Dropdown

pricing_program.providerid references supplychain.suppliers (see model codes table: {field: "providerid", table: "supplychain.suppliers", valuefield: "id", textfield: "supplier_name"}). The CRUD JSON MUST have a matching browserfields.alters entry — without it, the field renders as plain text with no dropdown:

"browserfields": {
    "exclouded": ["id", "ownerid"],
    "alters": {
        "providerid": {
            "uitype": "code",
            "dataurl": "{{entire_url('../api/get_search_providerid.dspy')}}",
            "valueField": "providerid",
            "textField": "providerid_text"
        }
    }
}

The supporting dspy (get_search_providerid.dspy) queries suppliers table via get_sor_context(request._run_ns, 'rbac') (not the old organization table). Returns [{providerid: '', providerid_text: '全部'}, ...]. New dspy files need explicit RBAC permission + rolepermission entries — use python3 set_role_perm.py 'owner.superuser' '/pricing/api/get_search_providerid.dspy' and also INSERT into rolepermission for non-superuser roles. See references/providerid-dropdown-setup.md.

Pitfalls

  1. pricing_spec and discount columns REMOVED (2026-07) — The pricing_spec and discount columns on pricing_program have been deleted. All field definitions now come from pricing_data.fields in the YAML. Pricing no longer applies a separate discount factor — cost = amount directly. Production migration requires ALTER TABLE pricing_program DROP COLUMN pricing_spec, DROP COLUMN discount;.

  2. formula field is REMOVED in new format: In the new pricing data design, the formula field is no longer used. Pricing is calculated declaratively from unit_prices * usage / unit_value. Old data with formula still works via backward compatibility, but new pricing_data should NOT include formulas.

  3. Multiple matches returned: If multiple rules match, ALL are returned (not just the first); caller must handle

  4. CRITICAL — between operator zero-value trap: ~= means a < v <= b (exclusive lower bound). When ANY factor is 0, the filter fails → pricing item skipped → amount=0 billing failure. Use =~ (a <= v < b) for ranges starting at 0. Real incident: 80 records lost billing (19.18元) because cached_tokens: 0.0 ~= 10000000.0 evaluated 0.0 < 0 = FALSE. Bulk fix: UPDATE pricing_program_timing SET pricing_data = REPLACE(pricing_data, '~=', '=~') WHERE pricing_data LIKE '%~=%';. Always test with factor=0 after any pricing_data change.

  5. Timing chain (拉链表): When adding a new pricing_program_timing record, use add_pricing_program_timing() function. It sets expired_date='9999-12-31' on the new record and updates the previous latest record's expired_date to the new enabled_date. Never INSERT directly without this chain logic.

  6. factor type skipped: Fields with type: factor are NOT included in Excel templates (used as calculation factors only)

  7. pricing_data structure: Must be {fields: {...}, pricings: [...]} — missing either part raises an error

  8. Cache key is {ppid}.{date}: Cached by pricing program ID + business date; keeps last 2 days per ppid

  9. Excel column headers are labels: Import maps label back to spec key — label must match exactly

  10. Qwen image pricing reference: For Alibaba Cloud/DashScope qwen-image-2.0-pro and dated Qwen image models, see references/qwen-image-pricing.md. Key pattern: both text-to-image and image-edit charge by successful output image count (usage.image_count), not resolution; use model + operation matching and formula: price * image_count.

  11. Timing cache invalidation events may not include ppid: pricing_program_timing:d:after payloads can contain only the timing row id, especially after SQLor delete paths. Cache refresh code must resolve the pricing program ID defensively: prefer payload.ppid, then payload.old / old_row / old_data, then look up pricing_program_timing.id -> ppid while the row still exists. For deletes, register and handle pricing_program_timing:d:before to pre-cache timing_id -> ppid, then use that cache from d:after and clean it up. This depends on SQLor dispatching d:before before row deletion; if production behavior regresses, update SQLor before blaming pricing cache code.

  12. Corrupted existing pricing data: Existing pricing programs may contain stale model names (e.g., viduq4-turbo instead of viduq3-turbo). When adding pricing for new capabilities, create a new independent pricing_program with clean data rather than modifying existing ones. Update llm_api_map.ppid to point to the new pricing program.

  13. pricing_data is YAML, not JSON: pricing_program_timing.pricing_data is stored as a YAML string (parsed by yaml.safe_load), not JSON. Use yaml.dump(data, allow_unicode=True) to generate, then MySQL-escape single quotes and backslashes before INSERT.

  14. Test before deploy: Use POST /pricing/test_pricing_program.dspy with {"ppid": "...", "data": {...}} to validate pricing calculations. Error responses include the full YAML used for matching — use it to debug missing options or mismatched rules. See references/pricing-test-guide.md.

  15. CRITICAL: pricing_program_timing records MUST exist: get_ppid_pricing(ppid) queries pricing_program JOIN pricing_program_timing. If a pricing_program has zero timing rows, the query returns 0 rows and raises data not found. A newly created pricing_program always needs at least one timing record with valid enabled_date <= today < expired_date. Before testing, verify: GET /pricing/get_all_pricing_programs.dspy — check the timings array is non-empty.

  16. Empty pricing_data causes DictObject crash: If pricing_program_timing.pricing_data is an empty string, yaml.safe_load('') returns None, then DictObject(**None) crashes with argument after ** must be a mapping. Error signature: config_data={...} yamlstr='' data not found. Ensure pricing_data contains valid YAML with both fields and pricings keys.

  17. discount variable in formulas: The discount value from pricing_program.discount is available as a variable inside formula expressions. Use formula: base_rate * duration * discount rather than hardcoding the discount value. The engine multiplies amount * discount again for cost, so formulas that already include discount will double-apply it to cost (but amount is correct).

  18. Complete pricing test workflow: Before declaring pricing deployed, run through the full checklist in references/pricing-test-guide.md section "Complete Test Checklist". See the reference for the step-by-step procedure.

  19. Nested token detail fields in LLM usage data: Modern LLM APIs (Qwen/DashScope, OpenAI) return token breakdowns as nested dicts, NOT flat keys. The usage dict structure is:

    {
  "prompt_tokens": 89612,
  "completion_tokens": 286,
  "prompt_tokens_details": {"cached_tokens": 86912},
  "completion_tokens_details": {"reasoning_tokens": 92},
  "model": "qwen3.7-max"
}

    Formula field references must use dot notation for nested fields: prompt_tokens_details.cached_tokens (NOT cached_tokens), completion_tokens_details.reasoning_tokens (NOT reasoning_tokens). The engine uses eval() which handles dict['a.b'] only if the key literally contains a dot — but the actual data has nested dicts, so cached_tokens as a flat key will raise "没有(cached_tokens)数据". Always verify the exact usage dict structure from the provider's API response before writing formulas.

  20. pricing_program_timing for promotional periods: When a model has time-limited promotional pricing (e.g., half-price until a date, then full price), create TWO timing records with non-overlapping date ranges: one for the promo period and one starting from the promo end date with regular pricing. Both records must be in pricing_program_timing — the engine filters by enabled_date <= today < expired_date.

  21. Distributor pricing belongs in supplychain module, NOT pricing: Multi-level distributor pricing (供应商→平台→一级→二级) is handled by the supplychain module, not pricing. Pricing module only defines the base/official price (供应商官宣价). Supplychain applies markup/discount chains on top. Never add distributor-specific pricing logic to the pricing module.

  22. price_factors is a single string, NOT an array: In the new pricing data design, each pricing item's price_factors is a single string (the factor name like "prompt_tokens", "duration", "flat"), not an array of dicts. Each pricing item represents ONE factor with its own unit_prices and unit. Multi-factor pricing (e.g., input + output tokens) is expressed as separate pricing items, not one item with multiple factors.

  23. fields must include price_factors, unit_prices, unit: The new pricing_data format requires these three fields in fields definition (plus optional min_amount). If any are missing, the engine raises "定价项中的xxx在fields中没有定义". Always include:

fields:
  price_factors:
    type: string
    role: factor
    label: "计价因子"
  unit_prices:
    type: float
    role: factor
    label: "单位定价"
  unit:
    type: string
    role: factor
    label: "计价单位"
  min_amount:
    type: float
    role: factor
    label: "最低消费"
    default: 0

  1. unit must reference unit_values keys: The unit field in each pricing item must be a key from the unit_values mapping (e.g., "百万", "秒", "次"), not a raw number. The engine looks up the divisor from unit_values[unit] to calculate unit_prices * usage / unit_value.

  2. Old data conversion pattern: When converting old pricing_data (with formula field) to the new format:

  • Single-factor formulas (0.5 * image_count) → {price_factors: "image_count", unit_prices: 0.5, unit: "张"}
  • Multi-factor formulas (a*x + b*y) → split into separate items per factor
  • Flat prices (formula: 1.35) → {price_factors: "flat", unit_prices: 1.35, unit: "次"}
  • Range conditions (prompt_tokens: 0 ~= 32000) → move to filters array
  • See references/pricing-data-migration.md for the conversion script

  1. Backward compatibility for old pricing data: Old data without pricing_type defaults to per_use. Old data without role on fields defaults to filter. Old data without price_factors — display API falls back to showing the raw formula. All three defaults ensure gradual migration.

  2. Usage/consumption data NEVER stored in pricing module: Pricing module is stateless — it defines prices and calculates amounts, but does NOT track usage, quotas, or consumption history. All usage state (monthly_usage, usage_record, subscription) belongs to the charging module. When bundle/credits products are used, charging calls pricing's calculation function to get the amount, then records the usage itself. Never add usage tracking tables or state to the pricing module.

  3. wwwroot symlink must be top-level: When deploying pricing module to Sage, the symlink must be at the module root level: ln -sfn ~/repos/pricing/wwwroot ~/repos/sage/wwwroot/pricing. Linking only subdirectories (e.g., api/) is insufficient — ahserver won't discover the module. Verify: ls -la ~/repos/sage/wwwroot/pricing should show a symlink to the full wwwroot directory.

  4. Two pricing_data formats coexist in production: Most pricing_program_timing.pricing_data is YAML dict format ({fields: {...}, pricings: [...]}). However, vidu video series uses JSON array format ([{model, resolution, formula: <number>, ...}]) where formula is already a flat price, not an expression. Code that parses pricing_data must handle both: try json.loads() first for array detection, fall back to yaml.safe_load() for dict format. See references/pricing-data-formula-parsing.md for details.

  5. Production pricing_data has common quality issues: When programmatically reading/writing pricing_data, normalize before parsing: (1) replace \r\n with \n, (2) replace Unicode smart quotes " " with ASCII ", (3) rstrip() each line to fix trailing spaces on keys like "duration: ", (4) fix 3-space to 2-space indentation under list items. These issues cause silent YAML parse failures that manifest as "pricing_data is None".

  6. filters uses OR logic, not AND: When a pricing item has filters array (interval pricing), the engine should match ANY filter item, not require ALL to match. Common implementation mistake:

# WRONG - requires all filters to match
for filter_item in filters:
    if not match(filter_item, usages):
        p_ok = False  # one mismatch fails the whole pricing

# CORRECT - any filter match is sufficient
filter_matched = False
for filter_item in filters:
    if match(filter_item, usages):
        filter_matched = True
        break
if not filter_matched:
    p_ok = False

Example: prompt_tokens: "0 ~ 32000" and "32000 ~ 128000" are mutually exclusive ranges — only one should match per request.

  1. Data conversion validation: After running convert_pricing_to_new_design.py, verify the generated SQL by:

  2. Checking /tmp/pricing_converted.json for _NEEDS_MANUAL_REVIEW flags

  3. Spot-checking 2-3 records with complex formulas to ensure correct decomposition

  4. Running test calculations with test_pricing_calc.py before applying to production All 35 production records converted successfully with zero manual review needed (commit 7200454).

  5. Range filters use value_mode: between, NOT filters array: Token/duration range conditions (e.g., prompt_tokens: "0 ~= 10000000") are top-level attributes on the pricing item, matched via check_value() with AND logic. They are NOT inside a filters array. The field definition must include value_mode: between. Each range gets its own pricing item. Example: two pricing items for low-range and high-range token pricing, each with prompt_tokens: "0 ~= 10M" or "10M ~". The filters array is only for categorical/spec matching (resolution, model variant) with OR logic.

  6. derived field dot-to-underscore conversion: When defining derived expressions with nested fields (e.g., derived: "prompt_tokens - prompt_tokens_details.cached_tokens"), the engine converts dot notation to underscore for Python eval. The eval environment is built by flattening nested dicts: config_data['prompt_tokens_details']['cached_tokens'] becomes eval_env['prompt_tokens_details_cached_tokens']. The expression itself also has dots replaced with underscores. This allows natural YAML syntax while working within Python's variable naming constraints.

  7. unit_prices must be per-unit, not per-token: When converting old pricing data, unit_prices values should be the price per unit (e.g., 6.0 元/百万tokens), NOT per-token (6.0e-06). The old format stored per-token prices in unit_prices with implicit /1000000 in the formula. The new format divides by unit_values[unit] automatically, so use the human-readable price (6.0) with unit: 百万.

  8. Backward compatibility detection in get_pricing_from_ymalstr: The calculation function must support both old (formula-based) and new (price_factors-based) formats. Detection logic:

is_new_format = p.get('price_factors') is not None and p.get('unit_prices') is not None
is_old_format = p.get('formula') is not None

if not is_new_format and not is_old_format:
    continue  # skip invalid entries

if is_new_format:
    # New calculation: unit_prices * usage / unit_values[unit]
    factor_name = p['price_factors']
    usage_value = config_data.get(factor_name)
    if usage_value is None:
        continue
    amount = p['unit_prices'] * float(usage_value) / unit_values.get(p.get('unit', '次'), 1)
elif is_old_format:
    # Old calculation: eval(formula)
    amount = eval(p['formula'], {}, config_data)

Key points:

  • New format requires BOTH price_factors AND unit_prices to be present
  • Old format requires formula to be present
  • If neither format is detected, skip the pricing item (don't crash)
  • New format handles filters with OR logic for tiered pricing
  • Old format uses eval() which can access nested fields via dot notation (e.g., usage.prompt_tokens)

  1. get_pricing_display must NOT multiply unit_price by unit_val for new format: New format unit_prices already stores the human-readable display price (e.g., 6.0 means 6.0 元/百万). Multiplying by unit_val (1000000) produces 6,000,000 — a catastrophic display bug. Same applies to tiered pricing: raw_tier_price is already display price, use directly.

  2. API responses: omit empty fields, filter redundant tiered — user demands zero noise: User explicitly corrected multiple times: empty fields like "filters": {}, "formula": "", "min_amount": 0 are confusing and unreadable. Rules:

  • Omit when empty: filters/filter_labels (not {"filters": {}}), formula (not "formula": ""), min_amount when 0
  • Tiered entries: only include when unit_prices != main unit_price — duplicates are meaningless noise
  • Tiered filters: remove internal params (value_mode, xxx_tokens range conditions) — only keep meaningful filters like model
  • display_text starts as empty list [] — items naturally become first lines (header 【...】定价: (...) was removed per user request)
  • Test before commit: verify actual output matches your description — user caught that I claimed to fix tiered duplicates but didn't verify

  1. Business semantics: filters vs tiered: In pricing_data, filters (item-level) means "applicability conditions" — which models/configs this pricing rule applies to. tiered (inside price_factors) means "price variations" — different prices for different conditions within the same rule. These are NOT interchangeable. Example: filters: {model: "qwen3-max"} means "this pricing is for qwen3-max", while tiered: [{model: "qwen3-max", unit_prices: 10}] would be wrong — tiered is for price differences, not applicability.

  2. Production filters is a LIST of single-key dicts, NOT direct fields: The actual database format stores filters as filters: [{model: xxx}, {resolution: yyy}, {duration: '1'}] — a list where each item is a single-key dict. Code extracting filters MUST handle both this list format AND direct fields on the pricing item. Pattern:

# Extract from p.items() (direct fields)
for k, v in p.items():
    if k in skip_keys: continue
    fdef = fields.get(k, {})
    if fdef.get('role', 'filter') == 'filter':
        filters[k] = v

# ALSO extract from p['filters'] list (production format)
raw_filters = p.get('filters')
if isinstance(raw_filters, list):
    for fi in raw_filters:
        if isinstance(fi, dict):
            for k, v in fi.items():
                filters[k] = v

  1. display_text shows filter conditions inline: Each pricing line includes its filter conditions: - 时长: 0.56 元/秒 [model=viduq3-turbo, resolution=1080p, off_peak=0]. This makes multi-condition pricing (like video generation with model+resolution+off_peak) readable at a glance.

  2. Shell escaping pitfall with API keys containing special characters: When API keys contain -, _, or other special characters, avoid passing them inline via curl or shell commands. Instead, write Python scripts to files and execute them. Example: key xxbM1asORWUTM7xc-lGC- causes SyntaxError: unterminated string literal when embedded in curl -H "Authorization: Bearer ${key}". Fix: write to .py file with api_key = 'xxbM1asORWUTM7xc-lGC-' and execute python3 script.py.

  3. get_pricing_display MUST pass model parameter to prevent cross-model price leakage: When multiple models share a single ppid (e.g., viduq3-pro, viduq3-turbo both use ppid sF2gcl7UeANKtnZv8hfvL), calling get_pricing_display(ppid) without the model filter returns ALL pricing items for ALL models. The model parameter does an exact-match filter on item['filters']['model'] or item['filter_labels']['模型']. Every caller must pass the model name: get_pricing_display(ppid, model=l.name). Affected callers: llmage/utils.py (get_llms_sort_by_provider, get_llms_by_catelog), llmage/v1/pricing/index.dspy. Also fix data: model names in pricing_data filters must match llm.name exactly (viduq3 → viduq3-pro).

  4. CRITICAL: Sage deploys require site-packages/ sync, not just pkgs/ git pull: Sage loads Python modules from ~/sage/py3/lib/python3.10/site-packages/<module>/, NOT from ~/sage/pkgs/<module>/. After git pull in pkgs/<module>/, you MUST cp the updated .py files to site-packages/<module>/ and restart Sage. .dspy and .ui files hot-reload from the wwwroot symlink (which points to pkgs), but .py files in site-packages require a manual copy + Sage restart. Symptom: code changes look correct on disk in pkgs/ but Sage returns old behavior — check site-packages/<module>/ first.

  5. model filter: three-modes for multi-model vs single-model ppids: get_pricing_display(ppid, model=name) filters pricing items by model name. Three behaviors:

  • Exact match: item['filters']['model'] == model → include
  • model_mappings match: model_mappings.get(item_model) == model → include (handles mapping tables like qwen3.7-max-2026-05-17: qwen3.7-max)
  • Fallback: if filtered_items is empty after filtering, fall back to ALL items. This handles single-model ppids where pricing_data doesn't label items with a model filter key.

Without the fallback, single-model ppids (no model field in filters) would show no pricing at all when model parameter is passed. The filter must be: if filtered_items: items = filtered_items, NOT items = filtered_items unconditionally.

  1. DB model names must match llm.name: When pricing_data uses model: viduq3 but the llm table has name: viduq3-pro, the model filter won't match. Fix: UPDATE pricing_program_timing SET pricing_data = REPLACE(pricing_data, 'model: viduq3\\n', 'model: viduq3-pro\\n'). Always verify model name consistency between pricing_data filters and llm.name.

  2. CRITICAL: filters can be dict OR list in YAML — BOTH formats crash if not handled: Production pricing_data uses TWO formats for the filters field:

  • List of dicts: filters: [{model: viduq3-pro}, {resolution: 1080p}, {off_peak: 0}]
  • Single dict: filters: {model: qwen3.6-plus, prompt_tokens: "-0.1 ~= 256000.0"} get_pricing_from_ymalstr has TWO for filter_item in p['filters']: loops (line ~625 for filter check, line ~676 for tiered pricing). When p['filters'] is a dict, iterating gives string keys, and filter_item.get(...) crashes with 'str' object has no attribute 'get'. Fix both locations:
raw_filters = p['filters']
if isinstance(raw_filters, dict):
    filter_items = [raw_filters]
else:
    filter_items = raw_filters
for filter_item in filter_items:
    if not isinstance(filter_item, dict):
        continue
    ...

Symptom: ppid='...', data={...}, 'str' object has no attribute 'get'. Common for qwen3.6-plus and other range-priced models where filters are a single dict with model + prompt_tokens range.

  1. model_mappings must be checked in get_pricing_from_ymalstr filter matching: When a pricing item has model: qwen3.7-max-2026-05-17 and the request data has model: qwen3.7-max, the filter needs model_mappings to resolve the match. Parse model_mappings = d.get('model_mappings', {}) from YAML, then in the filter check: if item_model == model or model_mappings.get(item_model) == model. This is separate from the display-side model_mappings support (pitfall 45) — both billing and display need it.

  2. fields must define ALL filter keys used in pricings -- add missing definitions to YAML, not code workarounds: When pricing_data has filters: {prompt_tokens: "0 ~= 256000"} but fields doesn't include a prompt_tokens definition, f = d.fields.get(fk) returns None and continue skips the range check. ALL range-based pricing items then pass unconditionally, charging multiple tiers simultaneously. The correct fix is DATA, not code: add the missing field definitions to the YAML fields section with value_mode: between. For example:

fields:
  prompt_tokens:
    type: int
    role: filter
    label: 输入Token
    value_mode: between
  completion_tokens:
    type: int
    role: filter
    label: 输出Token
    value_mode: between

Update the DB record: yaml.safe_load(pricing_data) -> add missing field defs -> yaml.dump() -> UPDATE pricing_program_timing. Then restart Sage. Do NOT add auto-detection code in get_pricing_from_ymalstr — it masks the data quality issue and will recur for each new field.

  1. NEVER put from X import Y inside an if-block: If DictObject is already imported at module top, an inline from appPublic.dictObject import DictObject inside an if-branch causes Python to treat DictObject as a local variable (shadowing the module-level import). Error: local variable 'DictObject' referenced before assignment. Always use the top-level import. If a function needs a module that might not exist, import at function top, not inside a conditional branch.

  2. KTV async services: one LLM per service pair (submit + status check), not separate LLMs: Services with both an apiname (submit) and query_apiname (status poll) need only ONE LLM record and ONE pricing_program. The query_apiname in llm_api_map enables query_task_status() in asyncinference.py to poll status without creating a new llmusage. Symptom: duplicate pricing programs like pp_ktv_realesrgan (submit) + pp_ktv_realesrgan_status (status-only). Fix: delete the status-only LLM + its pricing_program + pricing_program_timing, keep only the submit LLM with query_apiname set. Also check for missing synth-generate LLM (media endpoint exists but no LLM record — create one with query_apiname=synth-status, copy pricing_data from deleted synth-status).

  3. KTV media endpoints must NOT generate their own UUID IDs: 14 v1/media endpoints had import uuid; params_kw.transno = str(uuid.uuid4()).replace("-","") which creates 32-char hex IDs incompatible with getID()'s 21-char format. All inference paths go through inference() → _inference_generator() which calls getID() for IDs and only sets transno if not already present. Remove the custom UUID line entirely from all media endpoint .dspy files. Clean up the now-empty if not params_kw.transno: block too.

  4. Derived field paths are provider-specific — CHECK actual API response: Different LLM providers return cached_tokens at different nesting levels. OpenAI/Qwen: prompt_tokens_details.cached_tokens (nested). Kimi/Moonshot: usage.cached_tokens (top-level peer of prompt_tokens). Writing derived for the wrong provider causes silent-eval-to-zero — all tokens bill at uncached rate. Always inspect one real API response first.

Provider cached_tokens path derived expression
OpenAI / Qwen prompt_tokens_details.cached_tokens prompt_tokens - prompt_tokens_details.cached_tokens
Kimi / Moonshot usage.cached_tokens prompt_tokens - usage.cached_tokens
  1. Multi-currency billing setup: When pricing_program sets currency='USD', the billing chain needs exchange_rate table with real rates. Steps:
  • Create accounting database config entry in sage/conf/config.json pointing to same DB as sage (test user can't CREATE DATABASE)
  • Run DDL from accounting/scripts/multi_currency_migration.sql: currency table + exchange_rate table + INSERT base currencies and USD/CNY rates
  • Chain: llm_charging() reads pricing_program.currency → get_user_currency(userorgid) → get_exchange_rate(from, to, 'buy_rate') → convert_to_base(amount, currency, 'sell_rate')
  • Without exchange_rate table: get_exchange_rate returns 1.0 (no conversion), making USD pricing behave as CNY prices. Must restart Sage after adding exchange_rate.
  • Test: POST /pricing/test_pricing_program.dspy — amount should show USD pricing (not double-counted by exchange rate).

GET /pricing/api/get_pricing_display.dspy?ppid=xxx returns structured human-readable pricing data for frontend display. The endpoint calls get_pricing_display(ppid) which:

Also available as a v1 API endpoint: GET /llmage/v1/pricing?model=xxx in the llmage module. This wrapper looks up the model's ppid via llm_api_map (with isdefaultcatelog='1') and calls env.get_pricing_display(ppid). Use this when you have a model name instead of a ppid.

  1. Fetches the cached pricing data via get_ppid_pricing(ppid)
  2. Parses the YAML pricing_data
  3. Extracts filters (role=filter fields) with Chinese labels
  4. Extracts pricing items (each with price_factors, unit_prices, unit)
  5. Handles tiered pricing (filters with different unit_prices)
  6. Falls back to showing formula for old data without new format

Response structure (new format — clean, no empty fields):

{
  "status": "ok",
  "data": {
    "ppid": "...",
    "name": "通义千问 qwen3.7-max",
    "pricing_type": "per_use",
    "items": [
      {
        "price_factors": [{
          "factor": "uncache_tokens",
          "label": "非缓存tokens",
          "unit_price": 6.0,
          "unit": "百万",
          "unit_label": "元/百万"
        }]
      },
      {
        "price_factors": [{
          "factor": "cached_tokens",
          "label": "缓存tokens",
          "unit_price": 1.2,
          "unit": "百万",
          "unit_label": "元/百万"
        }]
      }
    ],
    "display_text": "【通义千问 qwen3.7-max】定价:\n  - 非缓存tokens: 6.0 元/百万\n  - 缓存tokens: 1.2 元/百万"
  }
}

Each item in items (fields omitted when empty/default):

{
  "filters": {"model": "MiniMax-M2.7"},          // omitted if empty
  "filter_labels": {"模型": "MiniMax-M2.7"},      // omitted if empty
  "price_factors": [
    {
      "factor": "prompt_tokens",
      "label": "输入Token",
      "unit_price": 2.1,
      "unit": "百万",
      "unit_label": "元/百万",
      "tiered": [                                  // only if price != main unit_price
        {"filters": {"model": "X"}, "unit_prices": 4.0}
      ]
    }
  ],
  "formula": "...",                                // omitted if empty
  "min_amount": 0.01                               // omitted if 0
}

Key implementation details:

  • Each item in items represents ONE pricing rule from the YAML
  • price_factors is an array with ONE element (the factor name)
  • tiered array only appears when price differs from main unit_price
  • filter_labels contains Chinese labels for display
  • display_text is a human-readable price table string (like an official pricing page)
  • Old format (with formula) returns formula field populated, price_factors as array of dicts
  • CRITICAL: For new format, unit_price is already the display price (e.g., 6.0 元/百万). Do NOT multiply by unit_val
  • CLEAN OUTPUT: Omit empty dicts (filters: {}), empty strings (formula: ""), zero defaults (min_amount: 0). Tiered entries with same price as main are redundant noise — filter them out. Internal params like value_mode, xxx_tokens range conditions are not user-facing — only keep meaningful filters like model

generate_formula_from_factors(price_factors) is DEPRECATED — the new format does not use formulas. Retained only for backward compatibility with old data.

Planned Functions (Not Yet Implemented)

These functions are part of the pricing_type expansion plan (charging module integration):

Function Purpose
calculate_bundle_usage(ppid, product, usage, month_used) Calculate quota consumption + overage cost for monthly_bundle
calculate_credits_usage(ppid, product, usage, remaining) Calculate credit deduction for monthly_credits