fix: address dev standards — README, SKILL.md, logging, docs
- Rewrite README: full install steps, usage flow, parser_config examples, builtin table, extractor types, status logic, known limitations - Add SKILL.md: architecture, adding parsers, deployment checklist, pitfalls - engine.py: replace bare except with traceback logging - parsers.py: CSS extractor docstring documents lightweight limitation
This commit is contained in:
parent
4a8f2c22be
commit
b9bd705b43
163
README.md
163
README.md
@ -1,35 +1,158 @@
|
|||||||
# 热点雷达 (Hotspot Radar)
|
# 热点雷达 (Hotspot Radar)
|
||||||
|
|
||||||
Sage 框架模块 — 指定来源的热点聚合分析平台。
|
Sage 框架模块 — 指定来源的热点聚合分析平台，支持动态添加来源。
|
||||||
|
|
||||||
- 11 个内置平台解析器（微博/知乎/百度/头条/B站/抖音/36氪/GitHub/HN/V2EX/RSSHub）
|
## 功能
|
||||||
- 4 种可配置提取器（json_path / css / regex / rss）
|
|
||||||
- 每条来源自带 parser_config，同类型不同来源可用不同提取规则
|
|
||||||
- 五维分析 + 7 天半衰期自动状态分类
|
|
||||||
|
|
||||||
## 目录
|
- **11 个内置平台解析器**：微博/知乎/百度/头条/B站/抖音/36氪/GitHub/HackerNews/V2EX/RSSHub
|
||||||
|
- **4 种可配置提取器**：json_path / css(轻量) / regex / rss
|
||||||
|
- **每条来源自带 parser_config**：JSON 格式的提取规则，同类型不同来源可用不同规则
|
||||||
|
- **五维分析**：时效 / 热度 / 内容 / 传播 / 受众，每维度 0-100 分
|
||||||
|
- **自动状态分类**：7 天半衰期指数衰减模型，emerging → rising → hot → cooling → expired
|
||||||
|
- **抓取日志**：每次执行一条审计记录（耗时/成功数/错误/HTTP状态码）
|
||||||
|
- **预警规则**：热度阈值 / 速度阈值 / 情感变化触发
|
||||||
|
|
||||||
|
## 目录结构
|
||||||
|
|
||||||
```
|
```
|
||||||
hotspot/
|
hotspot/
|
||||||
├── hotspot/ # Python 包（放入 Sage 根目录）
|
├── hotspot/ # Python 包
|
||||||
│ ├── __init__.py
|
│ ├── __init__.py # public API: run_fetch, run_analysis, get_stats
|
||||||
│ ├── engine.py # 抓取引擎
|
│ ├── engine.py # 抓取引擎 — 调度→fetch→extract→save→log
|
||||||
│ ├── analysis.py # 五维分析引擎
|
│ ├── analysis.py # 五维分析 + 衰减模型状态分类
|
||||||
│ ├── parsers.py # 11 内置解析器 + 通用提取器
|
│ ├── parsers.py # 11 内置解析器 + 4 通用提取器
|
||||||
│ └── db.py # 数据库操作
|
│ └── db.py # 数据库操作 (CRUD helpers)
|
||||||
├── json/ # 表定义（放入 Sage json/ 目录）
|
├── json/ # Sage 表定义 (6 张表)
|
||||||
├── wwwroot/hotspot/ # UI + 端点（放入 Sage wwwroot/hotspot/）
|
├── wwwroot/hotspot/ # Sage Web 层
|
||||||
├── ddl/mysql.sql # 建表 SQL
|
│ ├── index.ui # 仪表盘 UI (7 统计卡片 + 6 标签页)
|
||||||
└── install/ # 安装片段
|
│ ├── fetch_now.dspy # 触发抓取 (薄封装)
|
||||||
|
│ ├── analyze.dspy # 触发分析 (薄封装)
|
||||||
|
│ └── stats.dspy # 统计查询 (薄封装)
|
||||||
|
├── ddl/mysql.sql # 建表 SQL (6 张表 + 索引)
|
||||||
|
└── install/ # 部署片段
|
||||||
|
├── load_path_append.txt # 追加到 load_path.py 的路径
|
||||||
|
└── menu_append.json # 追加到 menu.ui 的菜单项
|
||||||
```
|
```
|
||||||
|
|
||||||
## 安装
|
## 安装
|
||||||
|
|
||||||
|
### 1. 复制文件到 Sage 部署目录
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
SAGE_ROOT=/path/to/sage
|
SAGE_ROOT=/path/to/sage
|
||||||
cp -r hotspot $SAGE_ROOT/ # Python 包
|
git clone git@git.opencomputing.cn:yumoqing/hotspot.git /tmp/hotspot
|
||||||
cp json/hotspot_*.json $SAGE_ROOT/json/
|
|
||||||
cp -r wwwroot/hotspot $SAGE_ROOT/wwwroot/
|
# Python 包
|
||||||
|
cp -r /tmp/hotspot/hotspot $SAGE_ROOT/
|
||||||
|
|
||||||
|
# 表定义
|
||||||
|
cp /tmp/hotspot/json/hotspot_*.json $SAGE_ROOT/json/
|
||||||
|
|
||||||
|
# Web UI + 端点
|
||||||
|
cp -r /tmp/hotspot/wwwroot/hotspot $SAGE_ROOT/wwwroot/
|
||||||
```
|
```
|
||||||
|
|
||||||
建表、注册权限、加菜单 → 见 README 原文。
|
### 2. 建表
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mysql -u <user> -p sage < /tmp/hotspot/ddl/mysql.sql
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. 注册权限
|
||||||
|
|
||||||
|
打开 `$SAGE_ROOT/load_path.py`，找到末尾的 `"""` 结束符，在它**前面**插入 `install/load_path_append.txt` 的内容。
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /tmp/hotspot
|
||||||
|
# 找到 SAGE_ROOT/load_path.py 中最后一个 """ 的前一行
|
||||||
|
# 把 install/load_path_append.txt 的内容粘贴进去
|
||||||
|
# 然后：
|
||||||
|
cd $SAGE_ROOT && python load_path.py
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. 添加菜单
|
||||||
|
|
||||||
|
打开 `$SAGE_ROOT/wwwroot/menu.ui`，在 `items` 数组末尾追加 `install/menu_append.json` 的内容（注意前面加逗号）。
|
||||||
|
|
||||||
|
### 5. 重启 Sage
|
||||||
|
|
||||||
|
访问 `/hotspot`。
|
||||||
|
|
||||||
|
## 使用流程
|
||||||
|
|
||||||
|
1. **添加来源** → 进入「来源管理」标签页，点击新增
|
||||||
|
- 类型选 `api` 配合 `parser=builtin` 使用内置解析器
|
||||||
|
- 或选 `rss`/`api` 配合 `parser=json_path` 自定义提取规则
|
||||||
|
2. **配置提取规则** → 在 `parser_config` 字段填入 JSON：
|
||||||
|
```json
|
||||||
|
{"parser": "builtin", "builtin_name": "weibo_hot"}
|
||||||
|
```
|
||||||
|
或自定义：
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"parser": "json_path",
|
||||||
|
"item_path": "$.data.list[*]",
|
||||||
|
"field_map": {
|
||||||
|
"title": "word",
|
||||||
|
"url": "link",
|
||||||
|
"heat_score": "hotValue"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
3. **配置调度** → 在「调度配置」标签页添加 cron 或间隔
|
||||||
|
4. **执行抓取** → 仪表盘点击「立即抓取全部来源」
|
||||||
|
5. **查看日志** → 「抓取日志」标签页检查成功率/错误详情
|
||||||
|
6. **执行分析** → 点击「执行分析」计算五维评分和状态
|
||||||
|
7. **浏览热点** → 「热点条目」标签页按状态筛选
|
||||||
|
8. **设置预警** → 「预警规则」标签页配置自动通知
|
||||||
|
|
||||||
|
## 内置解析器
|
||||||
|
|
||||||
|
| 解析器名 | 平台 | API 端点 | 备注 |
|
||||||
|
|---------|------|---------|------|
|
||||||
|
| `weibo_hot` | 微博热搜 | weibo.com/ajax/side/hotSearch | 需 UA 头 |
|
||||||
|
| `zhihu_hot` | 知乎热榜 | zhihu.com/api/v3/feed/topstory/hot-lists | — |
|
||||||
|
| `baidu_hot` | 百度热搜 | top.baidu.com/board | — |
|
||||||
|
| `toutiao_hot` | 头条热榜 | toutiao.com/hot-event/hot-board | 需 Referer |
|
||||||
|
| `bilibili_hot` | B站热门 | api.bilibili.com/x/web-interface/popular | — |
|
||||||
|
| `douyin_hot` | 抖音热点 | 第三方聚合接口 | 非官方，可能不稳定 |
|
||||||
|
| `36kr_hot` | 36氪热榜 | 36kr.com/api/search/list/info-flow/hot | — |
|
||||||
|
| `github_trending` | GitHub | api.github.com/search/repositories | — |
|
||||||
|
| `hackernews` | HackerNews | firebaseio.com (两步抓取) | — |
|
||||||
|
| `v2ex_hot` | V2EX | v2ex.com/api/topics/hot.json | — |
|
||||||
|
| `rsshub` | RSSHub | 来源 url 即 RSSHub 地址 | 解析 RSS/Atom |
|
||||||
|
|
||||||
|
## 提取器类型
|
||||||
|
|
||||||
|
| parser 值 | 适用场景 | 配置要点 |
|
||||||
|
|-----------|---------|---------|
|
||||||
|
| `builtin` | 使用内置解析器 | 设置 `builtin_name` |
|
||||||
|
| `json_path` | REST API 返回 JSON | `item_path` + `field_map` |
|
||||||
|
| `css` | HTML 页面 (轻量) | `item_selector` + `field_map` — 仅支持 `<a>` 标签提取，完整 CSS 需引入 BeautifulSoup |
|
||||||
|
| `regex` | 非结构化文本 | `item_regex` (命名捕获组) + `field_map` |
|
||||||
|
| `rss` | RSS/Atom feed | 自动解析，无需额外配置 |
|
||||||
|
|
||||||
|
## 状态分类逻辑
|
||||||
|
|
||||||
|
```
|
||||||
|
热度衰减: adjusted_heat = raw_heat × e^(-hours_alive / 168)
|
||||||
|
状态判定:
|
||||||
|
hours_alive > 336 (14天) → expired
|
||||||
|
336 > hours_alive > 168 → cooling
|
||||||
|
adjusted_heat > 5000 → hot
|
||||||
|
adjusted_heat > 500 → rising (velocity > 50) / emerging
|
||||||
|
adjusted_heat > 100 → rising (velocity > 100) / emerging
|
||||||
|
其他 → emerging
|
||||||
|
```
|
||||||
|
|
||||||
|
## 依赖
|
||||||
|
|
||||||
|
- **Sage** >= 0.0.1
|
||||||
|
- **aiohttp** (Python, 用于异步 HTTP 请求)
|
||||||
|
- **MySQL** / MariaDB
|
||||||
|
|
||||||
|
## 已知限制
|
||||||
|
|
||||||
|
- CSS 提取器为轻量正则实现，复杂页面结构建议使用 `json_path` 或 `builtin`
|
||||||
|
- 抖音解析器使用第三方聚合接口，官方 API 需企业资质
|
||||||
|
- 无 Headless Browser 集成（需单独部署 Playwright/Selenium）
|
||||||
|
|||||||
92
SKILL.md
Normal file
92
SKILL.md
Normal file
@ -0,0 +1,92 @@
|
|||||||
|
---
|
||||||
|
name: hotspot-radar
|
||||||
|
description: Build, extend, or troubleshoot the Hotspot Radar module for Sage — hotspot aggregation, parser config, analysis engine, and deployment.
|
||||||
|
category: devops
|
||||||
|
---
|
||||||
|
|
||||||
|
# Hotspot Radar Module
|
||||||
|
|
||||||
|
When the user works with or asks about the Hotspot Radar platform (热点雷达), load this skill.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
- **Repo**: `/Users/ymq/devops/hotspot` → `git@git.opencomputing.cn:yumoqing/hotspot.git`
|
||||||
|
- **Python package**: `hotspot/` with 4 modules:
|
||||||
|
- `parsers.py` — 11 builtin platform parsers + 4 extractors (json_path/css/regex/rss)
|
||||||
|
- `engine.py` — `run_fetch()` orchestrates source→fetch→extract→save→log
|
||||||
|
- `analysis.py` — `run_analysis()` 5-dimension scoring + 7-day half-life classifier
|
||||||
|
- `db.py` — `save_items()`, `write_fetch_log()`, `get_stats()`
|
||||||
|
- **Sage integration**: `.dspy` files are 1-line wrappers calling the Python package
|
||||||
|
- **Tables** (6): `hotspot_source`, `hotspot_schedule`, `hotspot_fetch_log`, `hotspot_item`, `hotspot_analysis`, `hotspot_alert`
|
||||||
|
- **DB name**: `sage` (matches Sage const.py)
|
||||||
|
|
||||||
|
## Adding a New Builtin Parser
|
||||||
|
|
||||||
|
Edit `hotspot/parsers.py`, add at module bottom:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from hotspot.parsers import _reg
|
||||||
|
_reg('new_platform',
|
||||||
|
'https://api.example.com/hot',
|
||||||
|
lambda data: [
|
||||||
|
{'title': item['name'], 'url': item['link'], 'heat_score': item['score']}
|
||||||
|
for item in data.get('list', [])
|
||||||
|
],
|
||||||
|
{'User-Agent': 'Mozilla/5.0'})
|
||||||
|
```
|
||||||
|
|
||||||
|
Then in Sage, add a source with:
|
||||||
|
```json
|
||||||
|
{"parser": "builtin", "builtin_name": "new_platform"}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Adding a Custom Extractor Rule
|
||||||
|
|
||||||
|
In Sage → 来源管理 → 编辑 → parser_config:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"parser": "json_path",
|
||||||
|
"item_path": "$.result.items[*]",
|
||||||
|
"field_map": {
|
||||||
|
"title": "name",
|
||||||
|
"url": "href",
|
||||||
|
"heat_score": "popularity",
|
||||||
|
"summary": "description",
|
||||||
|
"category": "type"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Deployment Checklist
|
||||||
|
|
||||||
|
1. `cp -r hotspot $SAGE_ROOT/`
|
||||||
|
2. `cp json/hotspot_*.json $SAGE_ROOT/json/`
|
||||||
|
3. `cp -r wwwroot/hotspot $SAGE_ROOT/wwwroot/`
|
||||||
|
4. Execute `ddl/mysql.sql` against Sage database
|
||||||
|
5. Append `install/load_path_append.txt` into `load_path.py` before closing `"""`
|
||||||
|
6. Append `install/menu_append.json` into `menu.ui` items array
|
||||||
|
7. `cd $SAGE_ROOT && python load_path.py`
|
||||||
|
8. Restart Sage
|
||||||
|
|
||||||
|
## Pitfalls
|
||||||
|
|
||||||
|
- **Sage import path**: The `hotspot/` Python package must be at `$SAGE_ROOT/hotspot/` so `from hotspot.engine import run_fetch` works in DSPY files.
|
||||||
|
- **DB name**: Always `sage` — never change the dbname in JSON table defs.
|
||||||
|
- **parser_config JSON**: Must be valid JSON. Malformed config → no items extracted → log shows `items_total: 0`.
|
||||||
|
- **aiohttp timeout**: All HTTP calls use 30s timeout. Slow sources will be logged as failed.
|
||||||
|
- **CSS extractor limitation**: Only extracts `<a href="...">text</a>` patterns. Full CSS needs BeautifulSoup4 (not included — Sage doesn't bundle it).
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /Users/ymq/devops/hotspot
|
||||||
|
python3 -c "
|
||||||
|
import ast, json
|
||||||
|
for f in ['hotspot/__init__.py','hotspot/parsers.py','hotspot/engine.py','hotspot/analysis.py','hotspot/db.py']:
|
||||||
|
ast.parse(open(f).read()); print(f'OK {f}')
|
||||||
|
for t in ['hotspot_source','hotspot_schedule','hotspot_fetch_log','hotspot_item','hotspot_analysis','hotspot_alert']:
|
||||||
|
json.load(open(f'json/{t}.json')); print(f'OK {t}')
|
||||||
|
print('All OK')
|
||||||
|
"
|
||||||
|
```
|
||||||
@ -101,7 +101,10 @@ async def fetch_and_extract(session, src):
|
|||||||
|
|
||||||
return [], resp.status, size
|
return [], resp.status, size
|
||||||
|
|
||||||
except Exception:
|
except Exception as e:
|
||||||
|
import traceback
|
||||||
|
print(f'[hotspot] fetch error for {src.get("name",src.get("id","?"))}: {e}')
|
||||||
|
traceback.print_exc()
|
||||||
return [], 0, 0
|
return [], 0, 0
|
||||||
|
|
||||||
|
|
||||||
@ -117,6 +120,9 @@ async def run_fetch():
|
|||||||
try:
|
try:
|
||||||
items, code, size = await fetch_and_extract(session, src)
|
items, code, size = await fetch_and_extract(session, src)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
|
import traceback
|
||||||
|
print(f'[hotspot] run_fetch error for {src.get("name",src["id"])}: {e}')
|
||||||
|
traceback.print_exc()
|
||||||
await write_fetch_log(src['id'], now, int((time.time() - t0) * 1000),
|
await write_fetch_log(src['id'], now, int((time.time() - t0) * 1000),
|
||||||
'failed', 0, 0, 0, str(e)[:1000])
|
'failed', 0, 0, 0, str(e)[:1000])
|
||||||
results['errors'] += 1
|
results['errors'] += 1
|
||||||
|
|||||||
@ -202,7 +202,12 @@ def _field(obj, path):
|
|||||||
return val
|
return val
|
||||||
|
|
||||||
def extract_css(html, config):
|
def extract_css(html, config):
|
||||||
"""CSS 选择器提取 (轻量正则版)"""
|
"""CSS 选择器提取 (轻量正则版 — 仅提取 <a href>text</a>)
|
||||||
|
|
||||||
|
限制: 不解析真正的 CSS 选择器语法，仅匹配 <a> 标签。
|
||||||
|
复杂页面结构建议使用 json_path 或 builtin 解析器。
|
||||||
|
完整 CSS 支持需引入 BeautifulSoup4 (Sage 未预装)。
|
||||||
|
"""
|
||||||
results = []
|
results = []
|
||||||
for m in re.finditer(
|
for m in re.finditer(
|
||||||
r'<a[^>]+href=["\']([^"\']+)["\'][^>]*>([^<]*)</a>',
|
r'<a[^>]+href=["\']([^"\']+)["\'][^>]*>([^<]*)</a>',
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user