pipeline_core/skills_library/all/rag-operations/SKILL.md

name	description	version
rag-operations	RAG ops: RBAC permissions, DSPY fixes, deployment.	1.0.0

RAG Operations

SSH target: rag@rag.opencomputing.cn, workdir: /d/rag/ragserver. DSPY files live under pkgs/rag/wwwroot/, UI files alongside them. Local repo: ~/work/rag/rag, remote: git@git.opencomputing.cn:yumoqing/rag.git.

RBAC Permission Setup (new .dspy → 200 OK)

A 403 on a RAG .dspy endpoint means the RBAC middleware rejected the request. The fix requires BOTH a permission table entry AND a rolepermission table entry linking it to roleid='any'. Either one missing = 403.

Step 0 — Confirm the DSPY file exists

ssh rag@rag.opencomputing.cn "find /d/rag/ragserver/pkgs/rag -name '<name>.dspy'"
# Files are under .../wwwroot/... not directly under pkgs/rag/

Step 1 — Diagnose both tables

Use Python heredoc (avoids shell escaping of % and quotes):

ssh rag@rag.opencomputing.cn "cd /d/rag/ragserver && PYTHONPATH='pkgs/sqlor:pkgs/ahserver:pkgs/apppublic' /d/rag/ragserver/py3/bin/python << 'PYEOF'
import sys, asyncio
sys.path.insert(0, 'pkgs/sqlor')
sys.path.insert(0, 'pkgs/ahserver')
sys.path.insert(0, 'pkgs/apppublic')
from appPublic.jsonConfig import getConfig
from sqlor.dbpools import DBPools
async def main():
    conf = getConfig('/d/rag/ragserver')
    db = DBPools(conf['databases'])
    async with db.sqlorContext('rag') as sor:
        # permission table — %% escapes % in LIKE (sqlor uses %-style formatting)
        rows = await sor.sqlExe(
            \"SELECT id, path, permtype FROM permission WHERE path LIKE '%%<name>%%'\",
            ns={})
        print('=== permission table ===')
        for r in rows: print(dict(r))
        if not rows: print('(NONE — needs INSERT)')

        # rolepermission table — column is 'permid' NOT 'permissionid'
        rows2 = await sor.sqlExe(
            \"SELECT * FROM rolepermission WHERE permid IN (SELECT id FROM permission WHERE path LIKE '%%<name>%%')\",
            ns={})
        print()
        print('=== rolepermission table ===')
        for r in rows2: print(dict(r))
        if not rows2: print('(NONE — THIS IS THE PROBLEM even if permission exists!)')
asyncio.run(main())
PYEOF"

Replace <name> with the DSPY filename stem, e.g. tag_options.

Step 2a — Insert permission (if missing)

ssh rag@rag.opencomputing.cn "cd /d/rag/ragserver && PYTHONPATH='pkgs/sqlor:pkgs/ahserver:pkgs/apppublic' /d/rag/ragserver/py3/bin/python << 'PYEOF'
import sys, asyncio
sys.path.insert(0, 'pkgs/sqlor')
sys.path.insert(0, 'pkgs/ahserver')
sys.path.insert(0, 'pkgs/apppublic')
from appPublic.jsonConfig import getConfig
from sqlor.dbpools import DBPools
async def main():
    conf = getConfig('/d/rag/ragserver')
    db = DBPools(conf['databases'])
    async with db.sqlorContext('rag') as sor:
        await sor.sqlExe(
            \"INSERT INTO permission (id, path, permtype) VALUES ('<perm_id>', '/rag/knowledge_bases_list/<name>.dspy', NULL)\",
            ns={})
        print('INSERTED permission: <perm_id>')
asyncio.run(main())
PYEOF"

Step 2b — Insert rolepermission (MANDATORY)

Pattern: roleid='any' grants access to all authenticated users. This is the most commonly missed step — permission exists but 403 persists because rolepermission is missing.

ssh rag@rag.opencomputing.cn "cd /d/rag/ragserver && PYTHONPATH='pkgs/sqlor:pkgs/ahserver:pkgs/apppublic' /d/rag/ragserver/py3/bin/python << 'PYEOF'
import sys, asyncio, uuid
sys.path.insert(0, 'pkgs/sqlor')
sys.path.insert(0, 'pkgs/ahserver')
sys.path.insert(0, 'pkgs/apppublic')
from appPublic.jsonConfig import getConfig
from sqlor.dbpools import DBPools
async def main():
    conf = getConfig('/d/rag/ragserver')
    db = DBPools(conf['databases'])
    async with db.sqlorContext('rag') as sor:
        rp_id = 'rp_' + uuid.uuid4().hex[:12]
        await sor.sqlExe(
            \"INSERT INTO rolepermission (id, roleid, permid) VALUES ('\" + rp_id + \"', 'any', '<perm_id>')\",
            ns={})
        print('INSERTED: id=' + rp_id + ', roleid=any, permid=<perm_id>')
        # Verify
        rows = await sor.sqlExe(
            \"SELECT * FROM rolepermission WHERE permid = '<perm_id>'\",
            ns={})
        for r in rows: print('VERIFY:', dict(r))
asyncio.run(main())
PYEOF"

Step 3 — Restart ragserver (RBAC is loaded at startup)

ssh rag@rag.opencomputing.cn "ps aux | grep 'app/ragserver' | grep -v grep | awk '{print \$2}' | xargs kill; sleep 2; cd /d/rag/ragserver && nohup py3/bin/python app/ragserver.py > /dev/null 2>&1 &"

Step 4 — Verify (expect 200)

curl -s -o /dev/null -w '%{http_code}' 'https://rag.opencomputing.cn/rag/knowledge_bases_list/<name>.dspy?_webbricks_=1&_lang=zh'

RAG code deployment workflow

Standard flow for RAG module changes:

# Local: edit files under ~/work/rag/rag/
cd ~/work/rag/rag
git add <files>
git commit -m "description"
git push

# Server: pull. DSPY files auto-reload (hot_reload=True), no restart needed.
ssh rag@rag.opencomputing.cn "cd /d/rag/ragserver/pkgs/rag && git pull"

# Verify with curl

SSH background start patterns (when restart is needed):

• nohup ... & — hangs SSH; avoid.
• setsid ... & disown — works but disown may timeout. Acceptable.
• ssh -f user@host 'cmd' — reliable background launch.
• background=true in Hermes terminal tool — best option within Hermes.

Schema reference

permission table

Column	Notes
id	Primary key, e.g. tag_options_perm_001
path	DSPY path, e.g. /rag/knowledge_bases_list/tag_options.dspy
permtype	Usually NULL for DSPY endpoints

rolepermission table

Column	Notes
id	Primary key, auto-generated
roleid	'any' for all authenticated users
permid	Foreign key to permission.id — column is permid not permissionid

Pitfalls

• sqlor LIKE % escaping: sqlor uses Python %-style string formatting, so literal % in SQL must be doubled: LIKE '%tag%' → LIKE '%%tag%%'. Single % → TypeError: not enough arguments for format string.
• sqlor IN (${ids}$) list expansion fails on rag server: The rag server's sqlor version does NOT auto-expand Python lists for IN clauses. sor.sqlExe("... IN (${ids}$)", {"ids": ["a","b"]}) → Illegal parameter data types varchar and row. Workaround: build comma-separated quoted string manually — id_list = ','.join(["'" + str(x) + "'" for x in doc_ids]) and concatenate into SQL. Pass ns={}.
• DSPY media_tags JOIN crashes: The standard WHERE mt.media_id IN (${ids}$) in media_cards.dspy triggers the sqlor list-expansion bug above. Wrap in try/except so cards still render without tags on failure.
• KB deletion misses cleanup: The original delete_kb.dspy only deleted DB rows — vector DB (VDB), graph DB, physical files, entities/entity_relations were NOT cleaned up. Physical files were never deleted because fp.startswith("/idfile/") never matched actual paths (which are /44/126/...). Add VDB delete via vectordb.opencomputing.net:10443/v1/delete, graph delete via localhost:9092/api/graph/delete, entity/relation cleanup, and fix file path matching to fp.startswith("/").
• DSPY voice query only matched audio files: media_cards.dspy originally queried by audio extensions (mp3/wav/m4a...) only. Videos (mp4) with extracted voiceprints needed metadata LIKE '%%voiceprint_status%%done%%' in the WHERE clause.
• DSPY debugging technique: When a .dspy returns generic error like "加载失败", temporarily replace except Exception: return error_widget with except Exception as e: return widget with str(e) to surface the real error. A single Python syntax error in one line of the DSPY string blocks the entire file execution.
• rolepermission column name: It's permid (not permissionid). Using permissionid → OperationalError: Unknown column 'permissionid'.
• 403 with permission existing: Always check rolepermission table next — this is the #1 cause of "permission exists but still 403".
• DSPY file location: Files live under pkgs/rag/wwwroot/... not pkgs/rag/.... Use find to locate.
• ${var}$ placeholder syntax: Works in DSPY files but NOT in direct sor.sqlExe() calls from Python scripts. In Python, use string formatting (f-strings or concatenation) for values and pass ns={}.
• Server process name: The ragserver runs as app/ragserver.py (not sage.py). Kill with pkill -f 'app/ragserver' or the awk pattern above.

Media search results

• Media URLs must use entire_url(): safe_url() only prepends /idfile; use entire_url(safe_url(path)) → full https://rag.opencomputing.cn/idfile/... URL. Without this, VideoPlayer/Image/Audio widgets get relative paths that may not resolve.
• Video playback: Use native <video> tag (via Html widget), NOT bricks VideoPlayer. The bricks component has height-collapse issues in search cards. "<video controls autoplay muted playsinline style=\"width:100%;max-height:400px\" src=\"...\">" works reliably.
• Position info display: Chunks can have position metadata (bbox, start_time, end_time) stored in document_chunks.metadata JSON column. Display 📍 (x1,y1)-(x2,y2) for face bounding boxes, ⏱ 3.2s → 8.5s for video/audio timestamps.
• Voiceprint API response: POST /voiceprint/extract/submit returns {"status":"SUCCEEDED","embedding":[...],"embedding_dim":192} — NO speakers field. Check embedding_dim > 0 for success, not speakers.
• Tag display: Tags stored in TWO places — media_tags table (from processing) AND documents.metadata.tags JSON array (from add_tag UI). Display logic must read both sources and deduplicate.
• DSPY boolean: Use Python True/False not JS true/false. "autoplay": true → NameError. JSON serialization converts Python bools automatically.