---
name: pricing-module
description: Development patterns, pricing engine logic, and API reference for the pricing module — Sage platform's YAML-driven pricing calculation engine
author: Hermes Agent
tags: [pricing, billing, yaml, excel, rules-engine, sage]
---

# Pricing Module Skill

## Overview

The pricing module is Sage's **configurable pricing engine** that calculates costs based on YAML-defined pricing rules. It supports multi-dimensional condition matching (between, in, =, >, <, >=, <=), formula evaluation, data mappings, time-based pricing periods, and bulk Excel import/export.

## Module Location

`~/repos/pricing/`

## Architecture

```
pricing_program (定价项目)
    │
    └── pricing_program_timing (定价项目时序) — 1:N, time-scoped, 拉链表
            │
            └── pricing_data (YAML) — {pricing_type, fields: {...}, pricings: [...]}
```

**Timing chain (拉链表)**: `pricing_program_timing` uses a chain/zipper pattern — each record has `enabled_date` → `expired_date`. When adding a new record via `add_pricing_program_timing()`:
- New record's `expired_date` is set to `9999-12-31`
- Previous latest record's `expired_date` is updated to the new `enabled_date`
- This creates a continuous timeline: `[old_start, new_start)` → `[new_start, 9999-12-31)`

### In Sage: llmage → pricing Flow

```
llm.ppid ──→ pricing_program.id
                │
                └── buffered_charging(ppid, usages)
                        │
                        ├── get_ppid_pricing(ppid) — fetch pricing_data (cached)
                        ├── get_pricing_from_ymalstr(usages, pricing_data) — match rules
                        └── return [amount, cost]  (no discount applied)
```

## Key Tables

| Table | Purpose | Key Fields |
|-------|---------|------------|
| **pricing_program** | Pricing project definition | id, name, ownerid, providerid, pricing_belong |
| **pricing_program_timing** | Time-scoped pricing periods (拉链表) | id, ppid, name, enabled_date, expired_date, pricing_data(YAML) |
| **pricing_item** | Individual pricing line items | id, pptid, name, formula, spec_values |

## ServerEnv Functions (via load_pricing())

Available in `.dspy` files via `globals()`:

| Function | Purpose |
|----------|---------|
| `buffered_charging(ppid, data)` | Calculate pricing with memory cache (primary method) |
| `pricing_program_charging(sor, ppid, data)` | Calculate pricing without cache (with sor cursor) |
| `get_pricing_program(ppid)` | Get pricing program definition |
| `load_pricing_data(pptid, webpath_xlsx)` | Load pricing data from Excel file |
| `write_pricing_patten(request, ppid)` | Generate Excel pricing template |
| `test_pricing(pptid, data)` | Test pricing calculation, return total amount |
| `generate_formula_from_factors(price_factors)` | Auto-generate formula string from price_factors array |
| `add_pricing_program_timing(env, sor, ns)` | **NEW**: Timing chain logic — sets expired_date, updates previous record |
| `get_pricing_display(ppid, model=None)` | Return structured human-readable pricing data for frontend display. `model` parameter filters items to a specific model name (exact match + model_mappings). Fallback: if filtering yields empty results, returns all items (for single-model ppids without model filter keys). |

## Pricing Types

Three pricing modes controlled by `pricing_type` in pricing_data (default: `per_use`):

### per_use — 按量计费（现有模式）
Standard pay-per-use. Each pricings entry has filters + price_factors + formula.

### monthly_bundle — 包月套餐
Contains multiple products with quotas and overage pricing:
```yaml
pricing_type: monthly_bundle
pricings:
  - name: 企业AI套餐
    plan_level: enterprise                # filter
    base_price:
      amount: 9999
      unit_label: 元/月
    products:
      - product: gpt-4
        label: GPT-4
        included: 1000000                 # monthly quota
        unit_label: Token/月
        overage:                          # overage has its own price_factors
          price_factors:
            - factor: tokens
              label: Token
              unit_price: 0.03
              unit_label: 元/千Token
          formula: 0.03 * float(tokens) / 1000.0
```
Usage is tracked by charge/accounting module. Each product use calls `calculate_bundle_usage()` which checks quota, calculates overage cost.

### monthly_credits — 包月额度
Buy a credit pool, products consume from it:
```yaml
pricing_type: monthly_credits
pricings:
  - name: AI消耗包
    credit_amount: 10000
    unit_label: 元
    products:
      - product: gpt-4
        label: GPT-4
        cost_per_unit: 0.03
        unit_label: 元/千Token
```

## Pricing Spec Format — REMOVED (2026-07)

The `pricing_spec` column on `pricing_program` has been removed. Field definitions now come exclusively from `pricing_data.fields` in the YAML, which is already self-describing. The `pricing_spec` was a redundant copy of the field definitions from pricing_data — removing it simplifies the data model and eliminates sync issues between the two sources.

```yaml
model:
  type: str
  label: "模型"
  role: filter                            # NEW: filter vs factor
  options:
    - "viduq3-pro"
    - "viduq3-turbo"

resolution:
  type: str
  label: "分辨率"
  role: filter
  options:
    - "1024p"
    - "720p"

duration:
  type: int
  label: "时长"
  role: filter

price:
  type: float
  label: 单价
  role: factor                            # calculation factor, not a filter
```

**Required**: `price` field with `type: float` (for per_use pricing).

**Field properties**:
- `type`: str, int, float, bool, factor
- `label`: Display name
- `role`: filter (default, used for matching/display) | factor (calculation only, hidden from filters)
- `options`: Dropdown options (also used for matching)
- `value_mode`: Match mode (=, between, in, >, >=, <, <=)

## Pricing Data Format (pricing_program_timing.pricing_data) — NEW DESIGN

The pricing data uses a **declarative per-factor structure**. Each pricing item is a single factor with its unit price and unit. Complex formulas are decomposed into independent pricing items.

```yaml
# 计价单位数值映射（只包含本条记录实际用到的单位）
unit_values:
  百万: 1000000
  秒: 1

# 字段定义（必须包含 price_factors, unit_prices, unit, min_amount）
fields:
  price_factors:
    type: string
    role: factor
    label: "计价因子"
  unit_prices:
    type: float
    role: factor
    label: "单位定价"
  unit:
    type: string
    role: factor
    label: "计价单位"
  min_amount:
    type: float
    role: factor
    label: "最低消费"
    default: 0
  
  # derived 字段：从原始 usage 数据计算衍生字段
  uncached_prompt_tokens:
    type: int
    role: factor
    label: "非缓存输入Token"
    derived: "prompt_tokens - prompt_tokens_details.cached_tokens"
  
  cached_tokens:
    type: int
    role: factor
    label: "缓存Token"
    derived: "prompt_tokens_details.cached_tokens"
  
  # filter fields with range conditions need value_mode: between
  prompt_tokens:
    type: int
    role: filter
    label: "输入Token数"
    value_mode: between
  completion_tokens:
    type: int
    role: factor
    label: "输出Token数"
    value_mode: between
  model:
    type: string
    role: filter
    label: "模型"

# 定价规则 — 每条一个 price_factors，无 formula
pricings:
  # 简单按量定价
  - price_factors: prompt_tokens
    unit_prices: 3.2
    unit: 百万
  - price_factors: completion_tokens
    unit_prices: 16.0
    unit: 百万

  # 区间定价：使用 value_mode: between + 顶层范围条件（NOT filters 数组）
  - price_factors: uncached_prompt_tokens
    unit_prices: 6.0
    unit: 百万
    model: qwen3.7-max
    prompt_tokens: "0 ~= 10000000"  # AND condition, not OR
  
  - price_factors: uncached_prompt_tokens
    unit_prices: 12.0
    unit: 百万
    model: qwen3.7-max
    prompt_tokens: "10000000 ~"  # 超过10M的更高价格

  # 规格定价（按分辨率等 filter 匹配，使用 filters 数组）
  - price_factors: duration
    unit: 秒
    filters:
      - SR: 720
        unit_prices: 0.9
      - SR: 1080
        unit_prices: 1.6
  ### unit_values: Per-record unit mapping (NEW DESIGN)

  The `unit_values` mapping in pricing_data should **only contain units actually used in that record's pricings**, not a global map of all possible units. This keeps each pricing_data self-contained and avoids confusion.

  **Example:**
  ```yaml
  # Record that only uses "秒" (seconds)
  unit_values:
    秒: 1
  fields:
    price_factors: {type: string, role: factor, label: "计价因子"}
    unit_prices: {type: float, role: factor, label: "单位定价"}
    unit: {type: string, role: factor, label: "计价单位"}
  pricings:
    - price_factors: duration
      unit_prices: 0.9
      unit: 秒
  ```

  **Common units in production data:**
  - `元/百万tokens` → 1000000 (LLM token pricing)
  - `秒` → 1 (video/audio duration)
  - `次` → 1 (per-use flat pricing)
  - `张` → 1 (image generation)

  **Conversion script pattern:** When migrating old data, scan all `pricings` items to collect unique `unit` values, then build a minimal `unit_values` map:

  ```python
  used_units = set()
  for p in new_pricings:
      u = p.get('unit', '')
      if u:
          used_units.add(u)

  # Map known units to their values
  UNIT_VALUES = {
      '元/百万tokens': 1000000,
      '秒': 1,
      '次': 1,
      '张': 1,
      # ... other known units
  }

  filtered_unit_values = {k: v for k, v in UNIT_VALUES.items() if k in used_units}
  ```

  ## Calculation Logic (pseudo-code)

  ```python
  total = 0
  for item in pricings:
      factor_name = item['price_factors']
      if factor_name == 'flat':
          total += item['unit_prices']
          continue

      usage_value = usages.get(factor_name, 0)
      unit_price = item.get('unit_prices')

      # Range-based pricing: top-level field with value_mode: between
      # e.g., prompt_tokens: "0 ~= 10000000" — matched via check_value()
      # These are AND conditions with other top-level fields
    
      # Spec-based pricing: filters array with OR logic
      if 'filters' in item:
          for f in item['filters']:
              if match_condition(f, usages):
                  unit_price = f['unit_prices']
                  break

      unit_val = unit_values.get(item['unit'], 1)
      amount = unit_price * usage_value / unit_val

      # Apply min_amount if defined
      if 'min_amount' in item:
          amount = max(amount, item['min_amount'])

      total += amount
  ```

  ## Derived Fields (NEW — 2026-06)

  Modern LLM APIs return nested token breakdowns (e.g., `prompt_tokens_details.cached_tokens`). The pricing engine supports **derived fields** that compute values from nested usage data:

  ```yaml
  fields:
    uncached_prompt_tokens:
      type: int
      role: factor
      label: "非缓存输入Token"
      derived: "prompt_tokens - prompt_tokens_details.cached_tokens"
  
    cached_tokens:
      type: int
      role: factor
      label: "缓存Token"
      derived: "prompt_tokens_details.cached_tokens"
  ```

  **Implementation:**
  - Engine evaluates `derived` expressions before processing pricings
  - Dot notation (`prompt_tokens_details.cached_tokens`) is converted to underscore (`prompt_tokens_details_cached_tokens`) for Python eval
  - Nested dict values are flattened: `config_data['prompt_tokens_details']['cached_tokens']` → `eval_env['prompt_tokens_details_cached_tokens']`
  - If derived expression fails, field defaults to 0

  **Use case:** When usage data has `{"prompt_tokens": 1075, "prompt_tokens_details": {"cached_tokens": 500}}`, the derived field `uncached_prompt_tokens` computes `1075 - 500 = 575`, which can then be priced separately from `cached_tokens`.

  See `references/pricing-derived-fields.md` for implementation details and testing patterns.

  ## Range Filters vs Spec Filters

  Two distinct patterns for conditional pricing:

  ### 1. Range Filters (value_mode: between)

  For token/duration ranges, use **top-level field attributes** with `value_mode: between`:

  ```yaml
  fields:
    prompt_tokens:
      type: int
      role: filter
      label: "输入Token"
      value_mode: between

  pricings:
    - price_factors: uncached_prompt_tokens
      unit_prices: 6.0
      unit: 百万
      prompt_tokens: "0 ~= 10000000"  # AND condition, matched via check_value()
  ```

  **Key:** Each pricing item with a range condition is a **separate entry**. The engine matches ALL top-level fields (AND logic).

  ### 2. Spec Filters (filters array)

  For categorical matching (resolution, model variants), use **filters array** with OR logic:

  ```yaml
  pricings:
    - price_factors: duration
      unit: 秒
      filters:
        - SR: 720
          unit_prices: 0.9
        - SR: 1080
          unit_prices: 1.6
  ```

  **Key:** The `filters` array items are **mutually exclusive** — only one matches per request (OR logic).

### Coverage Analysis

| Scenario | Supported | Example |
|----------|-----------|---------|
| Simple per-unit (token/duration) | ✅ | `unit_prices * usage / unit_value` |
| Range pricing (token brackets) | ✅ | filters by prompt_tokens range |
| Quality/spec pricing (resolution) | ✅ | filters by SR |
| Cache/non-cache separate pricing | ✅ | separate pricing items |
| Fixed fee (per-use flat) | ✅ | `price_factors: flat` |
| Minimum charge | ✅ | `min_amount` field |
| Cumulative tiered (first 10K free) | ❌ | current is "match bracket" not "cumulative" |

## Pricing Matching Rules

`get_pricing_from_ymalstr(config_data, yamlstr)` matches input data against pricing rules:

1. Iterates all `pricings` entries
2. For each rule, checks every field against input via `check_value()`:
   - `=` (default): exact match `data_value == spec_value`
   - `between`: range match (`0 ~ 100` means `0 <= value < 100`)
   - `in`: enumeration match (`gpt-4 gpt-3.5`)
   - `>`/`<`/>=`/`<=`: comparison
3. **All fields must match** for a rule to be selected
4. Executes `formula` via `eval()` to calculate `amount`
5. Returns **all matching rules** (not just the first one)

### Data Mappings

```yaml
model_mappings:
  "doubao-seed-2-0-pro-260215": "doubao-seed-2-0-pro"
```
Input values are mapped through `*_mappings` before matching.

## References

- `references/pricing-test-guide.md` — Step-by-step pricing test checklist
- `references/qwen-image-pricing.md` — Qwen image model pricing patterns
- `references/uapiio-bricks-format.md` — uapiio input_fields bricks 兼容格式及 data/response 字段规范
- `references/rbac-redis-cache.md` — rbac userperm.py Redis L2 缓存实现要点与调试经验
- `references/pricing-type-design.md` — Design decisions for pricing_type expansion (price_factors, monthly_bundle, monthly_credits, module boundaries with supplychain)
- `references/pricing-display-implementation.md` — Display API implementation details, mock-based testing pattern, deployment checklist
- `references/pricing-data-formula-parsing.md` — Formula pattern taxonomy, reverse-parsing approach, production data quality pitfalls, two coexisting formats (YAML dict vs JSON array)
- `templates/model-pricing.sql` — INSERT template for model pricing with cached/uncached token derived fields
- `references/pricing-data-migration.md` — Old-to-new conversion rules, filter propagation pitfalls, deployment steps
- `references/production-pricing-verification.md` — End-to-end verification workflow: pull llmusage via bugfix SQL API, join for ppid, recalculate amounts, compare with production billing
- `scripts/convert_all_pricing.py` — Reusable conversion script: handles JSON arrays, YAML dicts, flat pricing, range filter propagation

## Planned Enhancements (Partially Implemented)

### ✅ price_factors: Declarative pricing (IMPLEMENTED — new design)

The pricing data format has been redesigned to use declarative per-factor pricing instead of formula-based evaluation. See "Pricing Data Format" section above for the complete new structure.

**Key change from original plan**: Instead of `price_factors` being an array of dicts alongside `formula`, each pricing item IS a single factor with `unit_prices` and `unit`. No formula field needed — the engine calculates `unit_prices * usage / unit_value`.

### pricing_type: Three pricing modes

```yaml
pricing_type: per_use | monthly_bundle | monthly_credits
```

- **per_use** (default, current mode): match conditions → calculate from price_factors + unit_prices
- **monthly_bundle**: subscription with product quotas + overage pricing
- **monthly_credits**: buy credit balance, products consume from it

### role field in fields definition

```yaml
fields:
  model:
    type: str
    label: 模型
    role: filter     # filter = matching condition shown in "适用条件"
                     # factor = calculation factor (not shown as filter)
                     # default = filter (backward compat)
```

### monthly_bundle structure

```yaml
pricings:
  - name: 企业AI套餐
    base_price: {amount: 9999, unit_label: 元/月}
    products:
      - product: gpt-4
        label: GPT-4
        included: 1000000          # quota
        unit_label: Token/月
        overage:                   # overage has full price_factors
          price_factors:
            - factor: tokens
              label: Token
              unit_price: 0.03
              unit_label: 元/千Token
          formula: 0.03 * float(tokens) / 1000.0
```

### monthly_credits structure

```yaml
pricings:
  - name: AI消耗包
    credit_amount: 10000
    unit_label: 元
    products:
      - product: gpt-4
        label: GPT-4
        cost_per_unit: 0.03
        unit_label: 元/千Token
```

### Module separation: pricing vs charging

- **pricing module**: defines prices + calculates amounts (stateless, no consumption data)
- **charging module**: records consumption, tracks quotas, creates bills
  - subscription (主表): user_id, ppid, pricing_type, base_amount, dates, status
  - subscription_product (明细): sub_id, product, label, included, overage_config(JSON)
  - monthly_usage: sub_id, product, month, total_used, remaining, total_cost
  - usage_record: sub_id, product, usage_data, included_used, overage_used, amount

### Multi-level distribution belongs to supplychain module

Distribution layers (supplier → platform → level-1 → level-2) and markup/discount chains are NOT in pricing. Pricing only defines the base price (supplier list price). Supplychain module handles:
- Distributor hierarchy (parent_id tree)
- Per-level adjustment (discount/markup/fixed)
- Per-distributor price calculation (chain multiplication from base)

### Backward compatibility

- No pricing_type → default per_use
- No role → default filter
- No price_factors → display API falls back to showing formula
- New and old data coexist; migrate incrementally

## CRUD JSON Provider Dropdown

`pricing_program.providerid` references `supplychain.suppliers` (see model `codes` table: `{field: "providerid", table: "supplychain.suppliers", valuefield: "id", textfield: "supplier_name"}`). The CRUD JSON MUST have a matching `browserfields.alters` entry — without it, the field renders as plain text with no dropdown:

```json
"browserfields": {
    "exclouded": ["id", "ownerid"],
    "alters": {
        "providerid": {
            "uitype": "code",
            "dataurl": "{{entire_url('../api/get_search_providerid.dspy')}}",
            "valueField": "providerid",
            "textField": "providerid_text"
        }
    }
}
```

The supporting dspy (`get_search_providerid.dspy`) queries `suppliers` table via `get_sor_context(request._run_ns, 'rbac')` (not the old `organization` table). Returns `[{providerid: '', providerid_text: '全部'}, ...]`. **New dspy files need explicit RBAC `permission` + `rolepermission` entries** — use `python3 set_role_perm.py 'owner.superuser' '/pricing/api/get_search_providerid.dspy'` and also INSERT into `rolepermission` for non-superuser roles. See `references/providerid-dropdown-setup.md`.

## Pitfalls

1. **`pricing_spec` and `discount` columns REMOVED (2026-07)** — The `pricing_spec` and `discount` columns on `pricing_program` have been deleted. All field definitions now come from `pricing_data.fields` in the YAML. Pricing no longer applies a separate discount factor — cost = amount directly. Production migration requires `ALTER TABLE pricing_program DROP COLUMN pricing_spec, DROP COLUMN discount;`.
2. **formula field is REMOVED in new format**: In the new pricing data design, the `formula` field is no longer used. Pricing is calculated declaratively from `unit_prices * usage / unit_value`. Old data with `formula` still works via backward compatibility, but new pricing_data should NOT include formulas.
3. **Multiple matches returned**: If multiple rules match, ALL are returned (not just the first); caller must handle
4. **CRITICAL — between operator zero-value trap**: `~=` means `a < v <= b` (exclusive lower bound). When ANY factor is 0, the filter fails → pricing item skipped → **amount=0 billing failure**. Use `=~` (`a <= v < b`) for ranges starting at 0. Real incident: 80 records lost billing (19.18元) because `cached_tokens: 0.0 ~= 10000000.0` evaluated `0.0 < 0` = FALSE. **Bulk fix**: `UPDATE pricing_program_timing SET pricing_data = REPLACE(pricing_data, '~=', '=~') WHERE pricing_data LIKE '%~=%';`. Always test with factor=0 after any pricing_data change.
6. **Timing chain (拉链表)**: When adding a new `pricing_program_timing` record, use `add_pricing_program_timing()` function. It sets `expired_date='9999-12-31'` on the new record and updates the previous latest record's `expired_date` to the new `enabled_date`. Never INSERT directly without this chain logic.
7. **factor type skipped**: Fields with `type: factor` are NOT included in Excel templates (used as calculation factors only)
6. **pricing_data structure**: Must be `{fields: {...}, pricings: [...]}` — missing either part raises an error
7. **Cache key is `{ppid}.{date}`**: Cached by pricing program ID + business date; keeps last 2 days per ppid
8. **Excel column headers are labels**: Import maps label back to spec key — label must match exactly
9. **Qwen image pricing reference**: For Alibaba Cloud/DashScope `qwen-image-2.0-pro` and dated Qwen image models, see `references/qwen-image-pricing.md`. Key pattern: both text-to-image and image-edit charge by successful output image count (`usage.image_count`), not resolution; use `model + operation` matching and `formula: price * image_count`.
12. **Timing cache invalidation events may not include `ppid`**: `pricing_program_timing:d:after` payloads can contain only the timing row `id`, especially after SQLor delete paths. Cache refresh code must resolve the pricing program ID defensively: prefer `payload.ppid`, then `payload.old` / `old_row` / `old_data`, then look up `pricing_program_timing.id -> ppid` while the row still exists. For deletes, register and handle `pricing_program_timing:d:before` to pre-cache `timing_id -> ppid`, then use that cache from `d:after` and clean it up. This depends on SQLor dispatching `d:before` before row deletion; if production behavior regresses, update SQLor before blaming pricing cache code.
13. **Corrupted existing pricing data**: Existing pricing programs may contain stale model names (e.g., `viduq4-turbo` instead of `viduq3-turbo`). When adding pricing for new capabilities, create a **new independent pricing_program** with clean data rather than modifying existing ones. Update `llm_api_map.ppid` to point to the new pricing program.
14. **pricing_data is YAML, not JSON**: `pricing_program_timing.pricing_data` is stored as a YAML string (parsed by `yaml.safe_load`), not JSON. Use `yaml.dump(data, allow_unicode=True)` to generate, then MySQL-escape single quotes and backslashes before INSERT.
15. **Test before deploy**: Use `POST /pricing/test_pricing_program.dspy` with `{"ppid": "...", "data": {...}}` to validate pricing calculations. Error responses include the full YAML used for matching — use it to debug missing options or mismatched rules. See `references/pricing-test-guide.md`.
16. **CRITICAL: pricing_program_timing records MUST exist**: `get_ppid_pricing(ppid)` queries `pricing_program JOIN pricing_program_timing`. If a pricing_program has **zero timing rows**, the query returns 0 rows and raises `data not found`. A newly created pricing_program **always needs at least one timing record** with valid `enabled_date <= today < expired_date`. Before testing, verify: `GET /pricing/get_all_pricing_programs.dspy` — check the `timings` array is non-empty.
17. **Empty pricing_data causes DictObject crash**: If `pricing_program_timing.pricing_data` is an empty string, `yaml.safe_load('')` returns `None`, then `DictObject(**None)` crashes with `argument after ** must be a mapping`. Error signature: `config_data={...} yamlstr='' data not found`. Ensure `pricing_data` contains valid YAML with both `fields` and `pricings` keys.
18. **discount variable in formulas**: The `discount` value from `pricing_program.discount` is available as a variable inside formula expressions. Use `formula: base_rate * duration * discount` rather than hardcoding the discount value. The engine multiplies `amount * discount` again for `cost`, so formulas that already include `discount` will double-apply it to cost (but amount is correct).
19. **Complete pricing test workflow**: Before declaring pricing deployed, run through the full checklist in `references/pricing-test-guide.md` section "Complete Test Checklist". See the reference for the step-by-step procedure.
20. **Nested token detail fields in LLM usage data**: Modern LLM APIs (Qwen/DashScope, OpenAI) return token breakdowns as nested dicts, NOT flat keys. The usage dict structure is:
    ```
    {
      "prompt_tokens": 89612,
      "completion_tokens": 286,
      "prompt_tokens_details": {"cached_tokens": 86912},
      "completion_tokens_details": {"reasoning_tokens": 92},
      "model": "qwen3.7-max"
    }
    ```
    **Formula field references must use dot notation for nested fields**: `prompt_tokens_details.cached_tokens` (NOT `cached_tokens`), `completion_tokens_details.reasoning_tokens` (NOT `reasoning_tokens`). The engine uses `eval()` which handles `dict['a.b']` only if the key literally contains a dot — but the actual data has nested dicts, so `cached_tokens` as a flat key will raise "没有(cached_tokens)数据". Always verify the exact usage dict structure from the provider's API response before writing formulas.
21. **pricing_program_timing for promotional periods**: When a model has time-limited promotional pricing (e.g., half-price until a date, then full price), create TWO timing records with non-overlapping date ranges: one for the promo period and one starting from the promo end date with regular pricing. Both records must be in `pricing_program_timing` — the engine filters by `enabled_date <= today < expired_date`.

22. **Distributor pricing belongs in supplychain module, NOT pricing**: Multi-level distributor pricing (供应商→平台→一级→二级) is handled by the supplychain module, not pricing. Pricing module only defines the base/official price (供应商官宣价). Supplychain applies markup/discount chains on top. Never add distributor-specific pricing logic to the pricing module.

23. **price_factors is a single string, NOT an array**: In the new pricing data design, each pricing item's `price_factors` is a **single string** (the factor name like "prompt_tokens", "duration", "flat"), not an array of dicts. Each pricing item represents ONE factor with its own `unit_prices` and `unit`. Multi-factor pricing (e.g., input + output tokens) is expressed as **separate pricing items**, not one item with multiple factors.

29. **fields must include price_factors, unit_prices, unit**: The new pricing_data format requires these three fields in `fields` definition (plus optional `min_amount`). If any are missing, the engine raises "定价项中的xxx在fields中没有定义". Always include:
```yaml
fields:
  price_factors:
    type: string
    role: factor
    label: "计价因子"
  unit_prices:
    type: float
    role: factor
    label: "单位定价"
  unit:
    type: string
    role: factor
    label: "计价单位"
  min_amount:
    type: float
    role: factor
    label: "最低消费"
    default: 0
```

30. **unit must reference unit_values keys**: The `unit` field in each pricing item must be a key from the `unit_values` mapping (e.g., "百万", "秒", "次"), not a raw number. The engine looks up the divisor from `unit_values[unit]` to calculate `unit_prices * usage / unit_value`.

31. **Old data conversion pattern**: When converting old pricing_data (with `formula` field) to the new format:
- Single-factor formulas (`0.5 * image_count`) → `{price_factors: "image_count", unit_prices: 0.5, unit: "张"}`
- Multi-factor formulas (`a*x + b*y`) → split into separate items per factor
- Flat prices (`formula: 1.35`) → `{price_factors: "flat", unit_prices: 1.35, unit: "次"}`
- Range conditions (`prompt_tokens: 0 ~= 32000`) → move to `filters` array
- See `references/pricing-data-migration.md` for the conversion script

24. **Backward compatibility for old pricing data**: Old data without `pricing_type` defaults to `per_use`. Old data without `role` on fields defaults to `filter`. Old data without `price_factors` — display API falls back to showing the raw formula. All three defaults ensure gradual migration.

25. **Usage/consumption data NEVER stored in pricing module**: Pricing module is stateless — it defines prices and calculates amounts, but does NOT track usage, quotas, or consumption history. All usage state (monthly_usage, usage_record, subscription) belongs to the **charging** module. When bundle/credits products are used, charging calls pricing's calculation function to get the amount, then records the usage itself. Never add usage tracking tables or state to the pricing module.

26. **wwwroot symlink must be top-level**: When deploying pricing module to Sage, the symlink must be at the module root level: `ln -sfn ~/repos/pricing/wwwroot ~/repos/sage/wwwroot/pricing`. Linking only subdirectories (e.g., `api/`) is insufficient — ahserver won't discover the module. Verify: `ls -la ~/repos/sage/wwwroot/pricing` should show a symlink to the full wwwroot directory.

27. **Two pricing_data formats coexist in production**: Most `pricing_program_timing.pricing_data` is YAML dict format (`{fields: {...}, pricings: [...]}`). However, vidu video series uses **JSON array** format (`[{model, resolution, formula: <number>, ...}]`) where formula is already a flat price, not an expression. Code that parses pricing_data must handle both: try `json.loads()` first for array detection, fall back to `yaml.safe_load()` for dict format. See `references/pricing-data-formula-parsing.md` for details.

28. **Production pricing_data has common quality issues**: When programmatically reading/writing pricing_data, normalize before parsing: (1) replace `\r\n` with `\n`, (2) replace Unicode smart quotes `"` `"` with ASCII `"`, (3) `rstrip()` each line to fix trailing spaces on keys like `"duration: "`, (4) fix 3-space to 2-space indentation under list items. These issues cause silent YAML parse failures that manifest as "pricing_data is None".

29. **filters uses OR logic, not AND**: When a pricing item has `filters` array (interval pricing), the engine should match **ANY** filter item, not require ALL to match. Common implementation mistake:
```python
# WRONG - requires all filters to match
for filter_item in filters:
    if not match(filter_item, usages):
        p_ok = False  # one mismatch fails the whole pricing

# CORRECT - any filter match is sufficient
filter_matched = False
for filter_item in filters:
    if match(filter_item, usages):
        filter_matched = True
        break
if not filter_matched:
    p_ok = False
```
Example: `prompt_tokens: "0 ~ 32000"` and `"32000 ~ 128000"` are mutually exclusive ranges — only one should match per request.

30. **Data conversion validation**: After running `convert_pricing_to_new_design.py`, verify the generated SQL by:
1. Checking `/tmp/pricing_converted.json` for `_NEEDS_MANUAL_REVIEW` flags
2. Spot-checking 2-3 records with complex formulas to ensure correct decomposition
3. Running test calculations with `test_pricing_calc.py` before applying to production
All 35 production records converted successfully with zero manual review needed (commit `7200454`).

32. **Range filters use value_mode: between, NOT filters array**: Token/duration range conditions (e.g., `prompt_tokens: "0 ~= 10000000"`) are **top-level attributes** on the pricing item, matched via `check_value()` with AND logic. They are NOT inside a `filters` array. The field definition must include `value_mode: between`. Each range gets its own pricing item. Example: two pricing items for low-range and high-range token pricing, each with `prompt_tokens: "0 ~= 10M"` or `"10M ~"`. The `filters` array is only for categorical/spec matching (resolution, model variant) with OR logic.

33. **derived field dot-to-underscore conversion**: When defining derived expressions with nested fields (e.g., `derived: "prompt_tokens - prompt_tokens_details.cached_tokens"`), the engine converts dot notation to underscore for Python eval. The eval environment is built by flattening nested dicts: `config_data['prompt_tokens_details']['cached_tokens']` becomes `eval_env['prompt_tokens_details_cached_tokens']`. The expression itself also has dots replaced with underscores. This allows natural YAML syntax while working within Python's variable naming constraints.

34. **unit_prices must be per-unit, not per-token**: When converting old pricing data, `unit_prices` values should be the price per unit (e.g., 6.0 元/百万tokens), NOT per-token (6.0e-06). The old format stored per-token prices in `unit_prices` with implicit `/1000000` in the formula. The new format divides by `unit_values[unit]` automatically, so use the human-readable price (6.0) with `unit: 百万`.

31. **Backward compatibility detection in get_pricing_from_ymalstr**: The calculation function must support both old (formula-based) and new (price_factors-based) formats. Detection logic:
```python
is_new_format = p.get('price_factors') is not None and p.get('unit_prices') is not None
is_old_format = p.get('formula') is not None

if not is_new_format and not is_old_format:
    continue  # skip invalid entries

if is_new_format:
    # New calculation: unit_prices * usage / unit_values[unit]
    factor_name = p['price_factors']
    usage_value = config_data.get(factor_name)
    if usage_value is None:
        continue
    amount = p['unit_prices'] * float(usage_value) / unit_values.get(p.get('unit', '次'), 1)
elif is_old_format:
    # Old calculation: eval(formula)
    amount = eval(p['formula'], {}, config_data)
```
**Key points:**
- New format requires BOTH `price_factors` AND `unit_prices` to be present
- Old format requires `formula` to be present
- If neither format is detected, skip the pricing item (don't crash)
- New format handles `filters` with OR logic for tiered pricing
- Old format uses `eval()` which can access nested fields via dot notation (e.g., `usage.prompt_tokens`)

35. **get_pricing_display must NOT multiply unit_price by unit_val for new format**: New format `unit_prices` already stores the human-readable display price (e.g., 6.0 means 6.0 元/百万). Multiplying by `unit_val` (1000000) produces 6,000,000 — a catastrophic display bug. Same applies to `tiered` pricing: `raw_tier_price` is already display price, use directly.

36. **API responses: omit empty fields, filter redundant tiered — user demands zero noise**: User explicitly corrected multiple times: empty fields like `"filters": {}`, `"formula": ""`, `"min_amount": 0` are confusing and unreadable. Rules:
- **Omit when empty**: `filters`/`filter_labels` (not `{"filters": {}}`), `formula` (not `"formula": ""`), `min_amount` when 0
- **Tiered entries**: only include when `unit_prices != main unit_price` — duplicates are meaningless noise
- **Tiered filters**: remove internal params (`value_mode`, `xxx_tokens` range conditions) — only keep meaningful filters like `model`
- **display_text** starts as empty list `[]` — items naturally become first lines (header `【...】定价: (...)` was removed per user request)
- **Test before commit**: verify actual output matches your description — user caught that I claimed to fix tiered duplicates but didn't verify

37. **Business semantics: filters vs tiered**: In pricing_data, `filters` (item-level) means "applicability conditions" — which models/configs this pricing rule applies to. `tiered` (inside price_factors) means "price variations" — different prices for different conditions within the same rule. These are NOT interchangeable. Example: `filters: {model: "qwen3-max"}` means "this pricing is for qwen3-max", while `tiered: [{model: "qwen3-max", unit_prices: 10}]` would be wrong — tiered is for price differences, not applicability.

38. **Production filters is a LIST of single-key dicts, NOT direct fields**: The actual database format stores filters as `filters: [{model: xxx}, {resolution: yyy}, {duration: '1'}]` — a list where each item is a single-key dict. Code extracting filters MUST handle both this list format AND direct fields on the pricing item. Pattern:
```python
# Extract from p.items() (direct fields)
for k, v in p.items():
    if k in skip_keys: continue
    fdef = fields.get(k, {})
    if fdef.get('role', 'filter') == 'filter':
        filters[k] = v

# ALSO extract from p['filters'] list (production format)
raw_filters = p.get('filters')
if isinstance(raw_filters, list):
    for fi in raw_filters:
        if isinstance(fi, dict):
            for k, v in fi.items():
                filters[k] = v
```

39. **display_text shows filter conditions inline**: Each pricing line includes its filter conditions: `  - 时长: 0.56 元/秒 [model=viduq3-turbo, resolution=1080p, off_peak=0]`. This makes multi-condition pricing (like video generation with model+resolution+off_peak) readable at a glance.

41. **Shell escaping pitfall with API keys containing special characters**: When API keys contain `-`, `_`, or other special characters, avoid passing them inline via curl or shell commands. Instead, write Python scripts to files and execute them. Example: key `xxbM1asORWUTM7xc-lGC-` causes `SyntaxError: unterminated string literal` when embedded in `curl -H "Authorization: Bearer ${key}"`. Fix: write to `.py` file with `api_key = 'xxbM1asORWUTM7xc-lGC-'` and execute `python3 script.py`.

43. **get_pricing_display MUST pass model parameter to prevent cross-model price leakage**: When multiple models share a single ppid (e.g., viduq3-pro, viduq3-turbo both use ppid `sF2gcl7UeANKtnZv8hfvL`), calling `get_pricing_display(ppid)` without the `model` filter returns ALL pricing items for ALL models. The `model` parameter does an exact-match filter on `item['filters']['model']` or `item['filter_labels']['模型']`. Every caller must pass the model name: `get_pricing_display(ppid, model=l.name)`. Affected callers: `llmage/utils.py` (get_llms_sort_by_provider, get_llms_by_catelog), `llmage/v1/pricing/index.dspy`. Also fix data: model names in pricing_data filters must match `llm.name` exactly (viduq3 → viduq3-pro).

44. **CRITICAL: Sage deploys require `site-packages/` sync, not just `pkgs/` git pull**: Sage loads Python modules from `~/sage/py3/lib/python3.10/site-packages/<module>/`, NOT from `~/sage/pkgs/<module>/`. After `git pull` in `pkgs/<module>/`, you MUST `cp` the updated `.py` files to `site-packages/<module>/` and restart Sage. `.dspy` and `.ui` files hot-reload from the wwwroot symlink (which points to pkgs), but `.py` files in site-packages require a manual copy + Sage restart. Symptom: code changes look correct on disk in `pkgs/` but Sage returns old behavior — check `site-packages/<module>/` first.

45. **model filter: three-modes for multi-model vs single-model ppids**: `get_pricing_display(ppid, model=name)` filters pricing items by model name. Three behaviors:
- **Exact match**: `item['filters']['model'] == model` → include
- **model_mappings match**: `model_mappings.get(item_model) == model` → include (handles mapping tables like `qwen3.7-max-2026-05-17: qwen3.7-max`)
- **Fallback**: if filtered_items is empty after filtering, fall back to ALL items. This handles single-model ppids where pricing_data doesn't label items with a `model` filter key.

Without the fallback, single-model ppids (no `model` field in filters) would show no pricing at all when `model` parameter is passed. The filter must be: `if filtered_items: items = filtered_items`, NOT `items = filtered_items` unconditionally.

46. **DB model names must match llm.name**: When pricing_data uses `model: viduq3` but the `llm` table has `name: viduq3-pro`, the model filter won't match. Fix: `UPDATE pricing_program_timing SET pricing_data = REPLACE(pricing_data, 'model: viduq3\\n', 'model: viduq3-pro\\n')`. Always verify model name consistency between `pricing_data` filters and `llm.name`.

47. **CRITICAL: filters can be dict OR list in YAML — BOTH formats crash if not handled**: Production `pricing_data` uses TWO formats for the `filters` field:
- **List of dicts**: `filters: [{model: viduq3-pro}, {resolution: 1080p}, {off_peak: 0}]`
- **Single dict**: `filters: {model: qwen3.6-plus, prompt_tokens: "-0.1 ~= 256000.0"}`
`get_pricing_from_ymalstr` has TWO `for filter_item in p['filters']:` loops (line ~625 for filter check, line ~676 for tiered pricing). When `p['filters']` is a dict, iterating gives string keys, and `filter_item.get(...)` crashes with `'str' object has no attribute 'get'`. Fix both locations:
```python
raw_filters = p['filters']
if isinstance(raw_filters, dict):
    filter_items = [raw_filters]
else:
    filter_items = raw_filters
for filter_item in filter_items:
    if not isinstance(filter_item, dict):
        continue
    ...
```
Symptom: `ppid='...', data={...}, 'str' object has no attribute 'get'`. Common for qwen3.6-plus and other range-priced models where filters are a single dict with model + prompt_tokens range.

48. **model_mappings must be checked in get_pricing_from_ymalstr filter matching**: When a pricing item has `model: qwen3.7-max-2026-05-17` and the request data has `model: qwen3.7-max`, the filter needs `model_mappings` to resolve the match. Parse `model_mappings = d.get('model_mappings', {})` from YAML, then in the filter check: `if item_model == model or model_mappings.get(item_model) == model`. This is separate from the display-side model_mappings support (pitfall 45) — both billing and display need it.

49. **fields must define ALL filter keys used in `pricings` -- add missing definitions to YAML, not code workarounds**: When pricing_data has `filters: {prompt_tokens: "0 ~= 256000"}` but `fields` doesn't include a `prompt_tokens` definition, `f = d.fields.get(fk)` returns None and `continue` skips the range check. ALL range-based pricing items then pass unconditionally, charging multiple tiers simultaneously. **The correct fix is DATA, not code: add the missing field definitions to the YAML fields section with `value_mode: between`.** For example:

```yaml
fields:
  prompt_tokens:
    type: int
    role: filter
    label: 输入Token
    value_mode: between
  completion_tokens:
    type: int
    role: filter
    label: 输出Token
    value_mode: between
```

Update the DB record: `yaml.safe_load(pricing_data)` -> add missing field defs -> `yaml.dump()` -> UPDATE `pricing_program_timing`. Then restart Sage. Do NOT add auto-detection code in `get_pricing_from_ymalstr` — it masks the data quality issue and will recur for each new field.

50. **NEVER put `from X import Y` inside an if-block**: If `DictObject` is already imported at module top, an inline `from appPublic.dictObject import DictObject` inside an if-branch causes Python to treat `DictObject` as a local variable (shadowing the module-level import). Error: `local variable 'DictObject' referenced before assignment`. Always use the top-level import. If a function needs a module that might not exist, import at function top, not inside a conditional branch.

51. **KTV async services: one LLM per service pair (submit + status check), not separate LLMs**: Services with both an `apiname` (submit) and `query_apiname` (status poll) need only ONE LLM record and ONE pricing_program. The `query_apiname` in `llm_api_map` enables `query_task_status()` in `asyncinference.py` to poll status without creating a new llmusage. Symptom: duplicate pricing programs like `pp_ktv_realesrgan` (submit) + `pp_ktv_realesrgan_status` (status-only). Fix: delete the status-only LLM + its pricing_program + pricing_program_timing, keep only the submit LLM with `query_apiname` set. Also check for missing synth-generate LLM (media endpoint exists but no LLM record — create one with query_apiname=synth-status, copy pricing_data from deleted synth-status).

52. **KTV media endpoints must NOT generate their own UUID IDs**: 14 v1/media endpoints had `import uuid; params_kw.transno = str(uuid.uuid4()).replace("-","")` which creates 32-char hex IDs incompatible with `getID()`'s 21-char format. All inference paths go through `inference()` → `_inference_generator()` which calls `getID()` for IDs and only sets `transno` if not already present. Remove the custom UUID line entirely from all media endpoint `.dspy` files. Clean up the now-empty `if not params_kw.transno:` block too.

53. **Derived field paths are provider-specific — CHECK actual API response**: Different LLM providers return `cached_tokens` at different nesting levels. OpenAI/Qwen: `prompt_tokens_details.cached_tokens` (nested). Kimi/Moonshot: `usage.cached_tokens` (top-level peer of prompt_tokens). Writing derived for the wrong provider causes silent-eval-to-zero — all tokens bill at uncached rate. Always inspect one real API response first.

   | Provider | cached_tokens path | derived expression |
   |----------|-------------------|-------------------|
   | OpenAI / Qwen | `prompt_tokens_details.cached_tokens` | `prompt_tokens - prompt_tokens_details.cached_tokens` |
   | Kimi / Moonshot | `usage.cached_tokens` | `prompt_tokens - usage.cached_tokens` |

54. **Multi-currency billing setup**: When pricing_program sets `currency='USD'`, the billing chain needs `exchange_rate` table with real rates. Steps:
   - Create `accounting` database config entry in `sage/conf/config.json` pointing to same DB as `sage` (test user can't CREATE DATABASE)
   - Run DDL from `accounting/scripts/multi_currency_migration.sql`: `currency` table + `exchange_rate` table + INSERT base currencies and USD/CNY rates
   - Chain: `llm_charging()` reads `pricing_program.currency` → `get_user_currency(userorgid)` → `get_exchange_rate(from, to, 'buy_rate')` → `convert_to_base(amount, currency, 'sell_rate')`
   - Without exchange_rate table: `get_exchange_rate` returns 1.0 (no conversion), making USD pricing behave as CNY prices. Must restart Sage after adding exchange_rate.
   - Test: `POST /pricing/test_pricing_program.dspy` — amount should show USD pricing (not double-counted by exchange rate).

`GET /pricing/api/get_pricing_display.dspy?ppid=xxx` returns structured human-readable pricing data for frontend display. The endpoint calls `get_pricing_display(ppid)` which:

**Also available as a v1 API endpoint**: `GET /llmage/v1/pricing?model=xxx` in the llmage module. This wrapper looks up the model's `ppid` via `llm_api_map` (with `isdefaultcatelog='1'`) and calls `env.get_pricing_display(ppid)`. Use this when you have a model name instead of a ppid.

1. Fetches the cached pricing data via `get_ppid_pricing(ppid)`
2. Parses the YAML pricing_data
3. Extracts filters (role=filter fields) with Chinese labels
4. Extracts pricing items (each with price_factors, unit_prices, unit)
5. Handles tiered pricing (filters with different unit_prices)
6. Falls back to showing formula for old data without new format

**Response structure (new format — clean, no empty fields):**
```json
{
  "status": "ok",
  "data": {
    "ppid": "...",
    "name": "通义千问 qwen3.7-max",
    "pricing_type": "per_use",
    "items": [
      {
        "price_factors": [{
          "factor": "uncache_tokens",
          "label": "非缓存tokens",
          "unit_price": 6.0,
          "unit": "百万",
          "unit_label": "元/百万"
        }]
      },
      {
        "price_factors": [{
          "factor": "cached_tokens",
          "label": "缓存tokens",
          "unit_price": 1.2,
          "unit": "百万",
          "unit_label": "元/百万"
        }]
      }
    ],
    "display_text": "【通义千问 qwen3.7-max】定价:\n  - 非缓存tokens: 6.0 元/百万\n  - 缓存tokens: 1.2 元/百万"
  }
}
```

Each item in `items` (fields omitted when empty/default):
```json
{
  "filters": {"model": "MiniMax-M2.7"},          // omitted if empty
  "filter_labels": {"模型": "MiniMax-M2.7"},      // omitted if empty
  "price_factors": [
    {
      "factor": "prompt_tokens",
      "label": "输入Token",
      "unit_price": 2.1,
      "unit": "百万",
      "unit_label": "元/百万",
      "tiered": [                                  // only if price != main unit_price
        {"filters": {"model": "X"}, "unit_prices": 4.0}
      ]
    }
  ],
  "formula": "...",                                // omitted if empty
  "min_amount": 0.01                               // omitted if 0
}
```

**Key implementation details:**
- Each item in `items` represents ONE pricing rule from the YAML
- `price_factors` is an array with ONE element (the factor name)
- `tiered` array only appears when price differs from main `unit_price`
- `filter_labels` contains Chinese labels for display
- `display_text` is a human-readable price table string (like an official pricing page)
- Old format (with `formula`) returns `formula` field populated, `price_factors` as array of dicts
- **CRITICAL**: For new format, `unit_price` is already the display price (e.g., 6.0 元/百万). Do NOT multiply by `unit_val`
- **CLEAN OUTPUT**: Omit empty dicts (`filters: {}`), empty strings (`formula: ""`), zero defaults (`min_amount: 0`). Tiered entries with same price as main are redundant noise — filter them out. Internal params like `value_mode`, `xxx_tokens` range conditions are not user-facing — only keep meaningful filters like `model`

**`generate_formula_from_factors(price_factors)`** is DEPRECATED — the new format does not use formulas. Retained only for backward compatibility with old data.

## Planned Functions (Not Yet Implemented)

These functions are part of the pricing_type expansion plan (charging module integration):

| Function | Purpose |
|----------|---------|
| `calculate_bundle_usage(ppid, product, usage, month_used)` | Calculate quota consumption + overage cost for monthly_bundle |
| `calculate_credits_usage(ppid, product, usage, remaining)` | Calculate credit deduction for monthly_credits |