yumoqing/pipeline_core
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
pipeline_core/skills_library/all/gpu-server-services/SKILL.md

12 KiB
Raw Blame History

name description version tags trigger_conditions
gpu-server-services Complete map of GPU server services on opencomputing.net — ports, nginx routing, systemd units, and operational commands. 1.0.0
gpu-server
infrastructure
media
opencomputing
services
nginx
systemd
Working on opencomputing.net GPU server
Need to find which service runs on which port
Troubleshooting service failures or restarting the media pipeline
Planning new services that need to coexist with existing ones

GPU Server Services — opencomputing.net

Server Overview

  • Host: opencomputing.net (ymq@, passwordless SSH + sudo)
  • GPU: 8×RTX4090 24GB
  • Entry: nginx :10443 with SNI domain routing
  • Infra: Redis :6379, MySQL :3306
  • Disk: 664GB usable under /data/ymq/

Service Map (by port)

Currently Running (July 2026)

Port Service Dir Python Env Model Path Git Repo Notes
8886 VDB向量库 /data/ymq/vdb wan22-service/py3 /data/ymq/vdb/db/milvus.db yumoqing/vdb Milvus via ahserver. upsert/search/delete. Sole VDB — 8887 removed
9080 KTV媒体服务 /data/ymq/media-server vllm-0.8.5 /data/ymq/models/MahmoudAshraf/mms-300m-1130-forced-aligner (字幕对齐) + LLM(API,字幕校准) yumoqing/media-server Central media hub, systemd. Routes: /subtitle/ /calibrate/ /merge-video/ /ktv/
9081 歌曲评分 /data/ymq/songrate-service wan22-service/py3 评分模型(内置) yumoqing/songrate-service KTV quality scoring
9082 Real-ESRGAN超分 /data/ymq/realesrgan-service wan22-service/py3 /data/ymq/models/RealESRGAN_x2plus.pth yumoqing/realesrgan-service 3 workers
9083 Demucs音源分离 /data/ymq/demucs-service wan22-service/py3 /data/ymq/.cache/torch/hub/checkpoints/955717e8-8726e21a.th yumoqing/demucs-service 4 workers
9084 KTV合成 /data/ymq/ktv-synth-service wan22-service/py3 无ML模型(纯ffmpeg) yumoqing/ktv-synth-service ffmpeg视频拼接+音频混流
9085 RVC声音转换 /data/ymq/rvc-service venv /data/ymq/rvc-models/ RVC-Project/RVC-WebUI Voice cloning
9086 CLIP向量化 /data/ymq/clip_embedding vllm-0.8.5 /data/ymq/models/laion/CLIP-ViT-H-14-laion2B-s32B-b79K yumoqing/clip_embedding /api/embed
9087 声纹Embedding /share/ymq/run/voiceprint 内置venv /share/models/ecapa-tdnn yumoqing/voiceprint ECAPA-TDNN, GPU1, extract/verify submit+status. Replaced T2T on this port
9090 Reranker重排 /data/ymq/bge-reranker vllm-0.8.5 /data/ymq/models/BAAI/bge-reranker-v2-m3 yumoqing/bge-reranker /api/rerank
9091 人脸服务 /data/ymq/face-service wan22-service/py3 /data/ymq/.insightface/models/buffalo_l yumoqing/face-service InsightFace buffalo_l
9092 图数据库 /data/ymq/graph-service wan22-service/py3 Neo4j内嵌(内存图) yumoqing/graph-service graph CRUD
9093 NER实体识别 /data/ymq/ner-service venv /data/ymq/models/gliner-multitask-large-v0.5 yumoqing/ner-service GLiNER multilingual
9908 视频评估 /data/ymq/video-eval vllm-0.8.5 PSNR/SSIM/VMAF yumoqing/video-eval systemd
9925 FastWhisper ASR /data/ymq/asr-service aligner/py3 /data/ymq/models/deepdml/faster-whisper-large-v3-turbo-ct2 yumoqing/asr-service GPU 6
9926 VibeVoice ASR /share/ymq/run/vibevoice-asr vllm-0.8.5 /share/models/VibeVoice-ASR-7B yumoqing/vibevoice-asr 7B BF16, GPU 0, 人声分离
9997 Reranker模型层 /share/run/reranker vllm-0.8.5 /share/models/BAAI/bge-reranker-v2-m3 — Qwen3-Reranker-0.6B raw
11434 Ollama systemd system Ollama模型目录 — LLM server

Shared Python environments: /data/ymq/wan22-service (yumoqing/wan22-service) and /data/ymq/aligner (yumoqing/aligner).

Currently Down

Port Service Notes
9991 三元组抽取 nginx config exists, no process
9994 FastVLM nginx config exists, no process
9995 TTS语音合成 nginx config exists, no process
9087 T2T文本生成 nginx config removed — port taken by voiceprint
8887 VDB旧实例 Removed — systemctl disabled, process killed, nginx cleaned — 8886 sole VDB
9089-9106 vLLM qwen3 18 instances all down

Systemd Units

All under /etc/systemd/system/:

embedding.service  — Qwen3-Embedding-0.6B, WorkDir /share/run/embeddings
reranker.service   — Qwen3-Reranker-0.6B, WorkDir /share/run/reranker
entities.service   — WorkDir /share/run/entities
triples.service    — WorkDir /share/run/triples (Type=forking)
milvus.service     — WorkDir /share/run/milvus (Type=forking, TimeoutStartSec=300)
neo4j.service      — Failed (exit code 1)
vdb.service        — Failed, disabled
clip.service       — Inactive (killed TERM)
rag.service        — Inactive (dead)
qwen3.service      — vLLM (from /share/run/qwen3)
qwen3coder.service
gemma4.service
fvlm.service
fastwhisper.service — Auto-restart loop, fails with exit code 2
nvidia-asr.service
ollama.service      — Active
media-server.service — Active (only service confirmed running)
aligner.service
comfyui.service
m2m.service
subtitler.service
songrate.service
f5tts.service
video-eval.service
connection.service

Nginx Routing (all on :10443)

Nginx uses SNI-based virtual hosts. Each <name>.opencomputing.net maps to a backend.

Standalone Domain Services

Domain Backend Notes
vectordb.opencomputing.net localhost:8886 Sole VDB — /milvus/ route removed, all traffic → 8886
embedding.opencomputing.net localhost:9086 CLIP ViT-H-14
reranker.opencomputing.net localhost:9090 BGE reranker
graphdb.opencomputing.net localhost:9092 Neo4j graph
entities.opencomputing.net localhost:9093 GLiNER NER
ktv.opencomputing.net localhost:9080 + sub-paths KTV media hub
ollama.opencomputing.net localhost:11434 Ollama LLM
evaluate.opencomputing.net localhost:9908 Video eval

media.opencomputing.net — Consolidated Media Services

ALL media/KTV services are routed under sub-paths of media.opencomputing.net:

Sub-path Port Service
/face/ 9091 人脸服务
/asr/ 9925 FastWhisper ASR
/vibevoice/ 9926 VibeVoice人声ASR
/voiceprint/ 9087 声纹Embedding
/demucs/ 9083 Demucs音源分离
/realesrgan/ 9082 Real-ESRGAN超分
/synth/ 9084 KTV合成
/rvc/ 9085 RVC声音转换
/songrate/ 9081 歌曲评分
/video-eval/ 9908 视频评估

Removed

  • t2t.opencomputing.net — nginx config deleted, port 9087 taken by voiceprint

RAG Pipeline Architecture

Source: ~/rag-pipeline/. NOT running as a service — code only.

Pipeline flow (defined in pipeline.py):

ingest: chunk → embed(CLIP :9086) → store(VDB :8886) → extract entities(LLM) → store(Graph :9092)
search: embed query → hybrid retrieve(vector+graph RRF) → rerank(BGE :9090) → generate(LLM)

Plugin registry (~/rag-pipeline/plugins/registry.py):

  • embedding: CLIP ViT-H-14 (dim=1024), BGE-M3 (not deployed)
  • vdb: Milvus Lite (8886), Qdrant (not deployed)
  • graph: NetworkX (9092), FalkorDB (blocked)
  • reranker: BGE Reranker v2-m3 (9090)
  • face: InsightFace buffalo_l (dim=512, 9091)
  • chunker: recursive, sentence
  • retriever: hybrid (vector+graph+RRF), vector_only

API endpoints (port 9093 in code, but ner-service occupies this port):

  • /api/status, /api/ingest, /api/search, /api/pipelines, /api/plugins

Quick Diagnostics

Full inventory spreadsheet: ~/GPU_Services.xlsx (24 services, 20 running, 4 stopped — generated from ssh ymq@opencomputing.net). Columns: 端口, Base URL, 服务名称, 系统服务名, 运行路径, Python环境, 远端仓库, 模型路径(本地), API说明, 服务功能说明, 状态. All model paths are local absolute filesystem paths.

# Which services are running right now
ssh ymq@opencomputing.net "sudo systemctl list-units --type=service --state=running"

# All listening ports and their processes
ssh ymq@opencomputing.net "ss -tlnp | grep LISTEN"

# GPU memory usage
ssh ymq@opencomputing.net "nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv,noheader"

# Check a specific service
ssh ymq@opencomputing.net "curl -s http://localhost:9091/api/status"

Pitfalls

External access requires :10443 — port 443 times out

nginx exposes ONLY :10443 for external traffic (SNI-routed vhosts). Plain HTTPS (port 443) to any *.opencomputing.net service hangs until timeout (curl HTTP 000). Every client — DSPY ingestion code, scripts, tests — must use https://<svc>.opencomputing.net:10443/.... Audit with grep -n 'opencomputing.net' <files> | grep -v ':10443' (must return nothing). A bare except: pass around such calls hides the timeout completely — the classic symptom is downstream data silently missing (e.g. RAG chunks with empty vector_id). Some admin endpoints (e.g. VDB /v1/listcollections) may additionally 403 from non-whitelisted client IPs; call from rag.opencomputing.cn or the GPU server itself when that happens.

Port 9093 conflict

ner-service (GLiNER FastAPI) occupies port 9093, but rag-pipeline's code also assumes port 9093. If rag-pipeline is deployed, pick a different port or consolidate.

Model path mismatch for ner-service

GLiNER model expected at /mnt/disk0/yumoqing/models/gliner-multitask-large-v0.5 — this path does not exist. Model needs to be downloaded or symlinked to the actual model location under /data/ymq/models/.

Voiceprint (9087) POST returns 405 / multipart uploads fail

Symptoms: curl -X POST http://localhost:9087/extract/submit -F 'file=@audio.wav' returns 405 Method Not Allowed with Allow: GET,HEAD, OR returns {"error": "audio_file required", "kw": []} (params_kw empty).

Root causes (two separate issues):

  1. _allowed_methods not updated: aiohttp StaticResource.__init__ sets _allowed_methods = set(self._routes). ProcessorResource.__init__ updates _routes but not _allowed_methods. The dispatcher sees Allow: GET,HEAD. Fix: add self._allowed_methods = set(self._routes.keys()) after the last _routes.update().

  2. get_session_userinfo crashes: When auth middleware isn't installed, auth.get_auth(request) raises RuntimeError('auth_middleware not installed'). This crashes getPostData during multipart processing, so file uploads aren't saved to params_kw. Fix: wrap in try/except in auth_api.py.

See references/ecapa-tdnn-voiceprint.md for complete fix + recovery steps.

Voiceprint startup: nohup ... & hangs SSH

On the GPU server, nohup cmd & or setsid cmd & hangs the SSH connection. Use ssh -f ymq@opencomputing.net "cd /share/ymq/run/voiceprint && PYTHONPATH=... python3 ah.py -p 9087 >> logs/voiceprint.log 2>&1" instead. fastwhisper.service is in activating (auto-restart) with exit code 2. Its ExecStart points to /d/ymq/run/fastwhisper/py3/bin/python ah.py — but the home directory is /data/ymq, not /d/ymq. This path mismatch is likely the cause.

vLLM instances all down

Ports 9089-9106 are all unresponsive. The t2t nginx upstream still lists them. Start with the skill vllm-multi-instance-gpu for the correct startup procedure — critical to also kill orphan VLLM::EngineCore processes first.

Many services use /share/run/ not ~/

embedding, reranker, entities, triples, milvus, qwen3, qwen3coder all have their working directories under /share/run/ with start.sh/stop.sh scripts. The ~/ directories contain the same service types but as ahserver-based implementations — these may be newer replacements for the /share/run/ versions.