yumoqing
b9bd705b43
- Rewrite README: full install steps, usage flow, parser_config examples, builtin table, extractor types, status logic, known limitations - Add SKILL.md: architecture, adding parsers, deployment checklist, pitfalls - engine.py: replace bare except with traceback logging - parsers.py: CSS extractor docstring documents lightweight limitation
93 lines
3.2 KiB
Markdown
93 lines
3.2 KiB
Markdown
---
|
|
name: hotspot-radar
|
|
description: Build, extend, or troubleshoot the Hotspot Radar module for Sage — hotspot aggregation, parser config, analysis engine, and deployment.
|
|
category: devops
|
|
---
|
|
|
|
# Hotspot Radar Module
|
|
|
|
When the user works with or asks about the Hotspot Radar platform (热点雷达), load this skill.
|
|
|
|
## Architecture
|
|
|
|
- **Repo**: `/Users/ymq/devops/hotspot` → `git@git.opencomputing.cn:yumoqing/hotspot.git`
|
|
- **Python package**: `hotspot/` with 4 modules:
|
|
- `parsers.py` — 11 builtin platform parsers + 4 extractors (json_path/css/regex/rss)
|
|
- `engine.py` — `run_fetch()` orchestrates source→fetch→extract→save→log
|
|
- `analysis.py` — `run_analysis()` 5-dimension scoring + 7-day half-life classifier
|
|
- `db.py` — `save_items()`, `write_fetch_log()`, `get_stats()`
|
|
- **Sage integration**: `.dspy` files are 1-line wrappers calling the Python package
|
|
- **Tables** (6): `hotspot_source`, `hotspot_schedule`, `hotspot_fetch_log`, `hotspot_item`, `hotspot_analysis`, `hotspot_alert`
|
|
- **DB name**: `sage` (matches Sage const.py)
|
|
|
|
## Adding a New Builtin Parser
|
|
|
|
Edit `hotspot/parsers.py`, add at module bottom:
|
|
|
|
```python
|
|
from hotspot.parsers import _reg
|
|
_reg('new_platform',
|
|
'https://api.example.com/hot',
|
|
lambda data: [
|
|
{'title': item['name'], 'url': item['link'], 'heat_score': item['score']}
|
|
for item in data.get('list', [])
|
|
],
|
|
{'User-Agent': 'Mozilla/5.0'})
|
|
```
|
|
|
|
Then in Sage, add a source with:
|
|
```json
|
|
{"parser": "builtin", "builtin_name": "new_platform"}
|
|
```
|
|
|
|
## Adding a Custom Extractor Rule
|
|
|
|
In Sage → 来源管理 → 编辑 → parser_config:
|
|
|
|
```json
|
|
{
|
|
"parser": "json_path",
|
|
"item_path": "$.result.items[*]",
|
|
"field_map": {
|
|
"title": "name",
|
|
"url": "href",
|
|
"heat_score": "popularity",
|
|
"summary": "description",
|
|
"category": "type"
|
|
}
|
|
}
|
|
```
|
|
|
|
## Deployment Checklist
|
|
|
|
1. `cp -r hotspot $SAGE_ROOT/`
|
|
2. `cp json/hotspot_*.json $SAGE_ROOT/json/`
|
|
3. `cp -r wwwroot/hotspot $SAGE_ROOT/wwwroot/`
|
|
4. Execute `ddl/mysql.sql` against Sage database
|
|
5. Append `install/load_path_append.txt` into `load_path.py` before closing `"""`
|
|
6. Append `install/menu_append.json` into `menu.ui` items array
|
|
7. `cd $SAGE_ROOT && python load_path.py`
|
|
8. Restart Sage
|
|
|
|
## Pitfalls
|
|
|
|
- **Sage import path**: The `hotspot/` Python package must be at `$SAGE_ROOT/hotspot/` so `from hotspot.engine import run_fetch` works in DSPY files.
|
|
- **DB name**: Always `sage` — never change the dbname in JSON table defs.
|
|
- **parser_config JSON**: Must be valid JSON. Malformed config → no items extracted → log shows `items_total: 0`.
|
|
- **aiohttp timeout**: All HTTP calls use 30s timeout. Slow sources will be logged as failed.
|
|
- **CSS extractor limitation**: Only extracts `<a href="...">text</a>` patterns. Full CSS needs BeautifulSoup4 (not included — Sage doesn't bundle it).
|
|
|
|
## Verification
|
|
|
|
```bash
|
|
cd /Users/ymq/devops/hotspot
|
|
python3 -c "
|
|
import ast, json
|
|
for f in ['hotspot/__init__.py','hotspot/parsers.py','hotspot/engine.py','hotspot/analysis.py','hotspot/db.py']:
|
|
ast.parse(open(f).read()); print(f'OK {f}')
|
|
for t in ['hotspot_source','hotspot_schedule','hotspot_fetch_log','hotspot_item','hotspot_analysis','hotspot_alert']:
|
|
json.load(open(f'json/{t}.json')); print(f'OK {t}')
|
|
print('All OK')
|
|
"
|
|
```
|