pipeline_core/skills_library/all/remote-app-testing/SKILL.md

name	category	description	trigger
remote-app-testing	devops	SSH-based remote application testing workflow for deployed Hermes applications	When asked to test, verify, or diagnose issues on a remote deployed Hermes application

Remote Application Testing Workflow

Sage Test Servers

Server	SSH	Sage Path	DB	Notes
tokentest	apitest@120.48.168.15 (tokentest.opencomputing.cn)	/d/apitest/sage --port 9180, pkgs/	mysql -u test -ptest123 sage	SSH限流, Sage重启后等30s重连

Sage部署: git push → SSH git pull in pkgs/ → load_path.py → pkill sage.py → restart。

Overview

Test deployed Hermes applications on remote servers via SSH without direct console access. Uses Python + paramiko for SSH automation and curl for HTTP testing.

Prerequisites

import paramiko

SSH_HOST = "crm.opencomputing.cn"
SSH_USER = "ymq"
SSH_PASS = "password"

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(SSH_HOST, port=22, username=SSH_USER, password=SSH_PASS, timeout=15)

SSH Command Pattern

def run(ssh, cmd, timeout=15):
    stdin, stdout, stderr = ssh.exec_command(cmd, timeout=timeout)
    out = stdout.read().decode(errors="replace")
    err = stderr.read().decode(errors="replace")
    rc = stdout.channel.recv_exit_status()
    return out, err, rc

Critical: Always use ||| as delimiter between body and HTTP status code in curl tests:

curl_cmd = "curl -s -w '|||%{http_code}' -b /tmp/cookie.txt http://localhost:8080/path"
out, _, _ = run(ssh, curl_cmd, timeout=15)
parts = out.split("|||")
body = "|||".join(parts[:-1])
code = int(parts[-1].strip())

Why ||| not \n: curl's %{http_code} can be concatenated to the body without newline in some cases. Using ||| as delimiter ensures reliable separation.

Important curl syntax: When embedding curl commands in triple-quoted Python strings, use %{http_code} NOT %%{http_code} — the % is NOT a Python format specifier inside run(ssh, cmd) since cmd is a plain string. However, when using python3 -c '...' with single quotes, use %%{http_code} to escape for shell.

Standard Test Phases

Phase 1: Service Health

ps aux | grep app_name | grep -v grep
curl -s -w '|||%{http_code}' http://localhost:8080/
tail -20 /tmp/app.log

Phase 2: Public Pages (no auth required)

Test pages that should be accessible without login:

• Login page: /main/rbac/user/login.ui
• Register page: /main/rbac/user/register.ui Expected: HTTP 200

Phase 3: Authentication

curl -s -c /tmp/test_cookie.txt -X POST \
  http://localhost:8080/main/rbac/user/up_login.dspy \
  -d 'username=admin&password=pass123'

Check response contains login success widget (Message widget).

Phase 4: Authenticated Access

# First login to get cookie
curl -s -c /tmp/test_cookie.txt -X POST \
  http://localhost:8080/main/rbac/user/up_login.dspy \
  -d 'username=admin&password=pass123' > /dev/null

# Then test protected pages with cookie
curl -s -b /tmp/test_cookie.txt http://localhost:8080/main/module/page.ui

Expected: HTTP 200 for authorized pages

Phase 5: Access Control (negative tests)

# Without cookie, protected pages should return 401
curl -s -o /dev/null -w '|||%{http_code}' \
  http://localhost:8080/main/module/protected.dspy

Expected: HTTP 401

Phase 6: CRUD Operations

# Create
curl -s -b /tmp/cookie.txt -X POST \
  http://localhost:8080/main/module/entity/add_entity.dspy \
  -d 'field1=value1&field2=value2'

# Read
curl -s -b /tmp/cookie.txt \
  http://localhost:8080/main/module/entity/get_entity.dspy

# Update
curl -s -b /tmp/cookie.txt -X POST \
  http://localhost:8080/main/module/entity/update_entity.dspy \
  -d 'id=xxx&field1=newvalue'

# Delete
curl -s -b /tmp/cookie.txt -X POST \
  http://localhost:8080/main/module/entity/delete_entity.dspy \
  -d 'id=xxx'

Phase 7: Database State

mysql -u user -ppass -h db dbname -e "SELECT COUNT(*) FROM table"
mysql -u user -ppass -h db dbname -e "SELECT * FROM table LIMIT 5"

Phase 8: Server Logs

tail -50 /tmp/app.log

Service Restart Pattern

Pitfall: kill/pkill kills SSH session

On remote servers where the SSH session runs under the same user as the target processes, kill -9, pkill -f, or sudo pkill will often terminate the SSH connection itself (exit code 255). The restart commands that follow never execute.

Workaround A — detached background script:

# Write kill script, then run via nohup so it survives SSH disconnect
ssh user@host 'nohup bash -c "pkill -9 -f app.py; sleep 3" > /tmp/kill.log 2>&1 &'
sleep 6
ssh user@host 'pgrep -f app.py | wc -l'  # reconnect and verify

Workaround B — kill by PID with SIGTERM first:

PIDS=$(pgrep -f "app.py")
kill $PIDS 2>/dev/null      # SIGTERM — less likely to kill SSH
sleep 3
# Only SIGKILL if necessary
kill -9 $(pgrep -f "app.py") 2>/dev/null

Pitfall: code changes must be local-first

NEVER modify source files directly on the server. User explicitly corrected this. Always:

1. Modify locally in ~/repos/<module>/
2. git add + git commit + git push
3. SSH to server: cd pkgs/<module> && git pull origin main
4. If Python package: pip install -e . (editable) or copy to site-packages
5. Restart service

Server-side file edits are lost on next git pull and cause drift. Only exception: temporary debug logging (must revert before commit).

Standard restart (paramiko)

import time

# Kill
run(ssh, "pkill -f app_name.py 2>/dev/null || true")
time.sleep(2)

# Start
transport = ssh.get_transport()
chan = transport.open_session(timeout=10)
chan.exec_command("cd /path/to/app && source py3/bin/activate && export PYTHONPATH=/path/to/app && nohup python app/app_name.py --port 8080 > /tmp/app.log 2>&1 &")
time.sleep(3)
chan.close()

# Verify
for i in range(10):
    time.sleep(2)
    out, _, _ = run(ssh, "ps aux | grep app_name | grep -v grep")
    if out.strip():
        break

Stress / Concurrency Testing

Test at 3 levels: 30, 60, 100 concurrency. Each test runs for 60 seconds. Use an async Python script with aiohttp + asyncio.Semaphore.

Resource monitoring

Sample in parallel with requests:

• CPU: /proc/stat every 1s
• Network: /proc/net/dev every 2s
• Disk IO: /proc/diskstats every 2s

Metrics collected

• Total requests, success/fail counts, failure type breakdown (timeout vs HTTP errors)
• TTFB and total time: count, min, p50, p75, p90, p95, p99, max, avg
• CPU: min, max, avg, p50, p95
• Network: rx/tx KB/s avg and max
• Disk: read/write IOPS avg and max

Script workflow

1. Write stress script locally with all monitoring built in
2. scp to server
3. Run sequentially: 30 → 60 → 100 concurrency
4. Save JSON report per test to /tmp/stress_c{N}_v2.json
5. Compare across concurrency levels

Profiling before optimization

Before optimizing, instrument the hot path with timing probes to identify the actual bottleneck. Use a lightweight timing module that monkey-patches json.loads/dumps and provides @timed decorators for async functions. Collect per-request breakdown, then revert all instrumentation. Never profile in production — only in test environment with user consent.

Reference template: references/llm_stress_template.py — minimal async stress test script without resource monitors. Full version with CPU/network/disk monitoring at /d/ymq/llm_stress_v2.py.

Common Issues to Check

1. Service not running → Check process list, restart
2. 401 Unauthorized on public pages → Check anonymous/any role permissions in DB
3. 404 Not Found → Check wwwroot symlinks exist: ls -la wwwroot/
4. 500 Internal Server Error → Check server log tail, look for Python exceptions
5. Path resolution errors → Check ahserver url2file.py bug (prefix not stripped)
6. Permission not enforced → Check rbac userperm.py wildcard matching bug
7. True/false syntax errors → .dspy files using JavaScript true instead of Python True

File Upload Pattern

sftp = ssh.open_sftp()
sftp.put("/local/path/file.py", "/remote/path/file.py")
sftp.close()

Important URL Mapping Rule

In Hermes apps, URL paths map to filesystem as:

• Config: ["wwwroot", "/main"] means wwwroot/ serves at /main/
• URL /main/rbac/user/login.ui → Filesystem wwwroot/rbac/user/login.ui
• The /main prefix is STRIPPED before filesystem lookup

If symlinks exist in wwwroot pointing to module wwwroot directories:

• wwwroot/rbac → pkgs/rbac/wwwroot
• wwwroot/customer_management → pkgs/customer_management/wwwroot

Progressive Exploration Pattern

For multi-step deploy-test tasks, use a progressively deepening approach: explore first, then act. This avoids blindly executing steps that may already be satisfied.

Pattern: Sequential paramiko scripts with fresh connections

Rather than one monolithic script, write a series of small scripts, each focused on one phase, run sequentially:

script1.py   →  explore (what exists? what state?)
script2.py   →  check specific conditions
script3.py   →  act based on findings
script4.py   →  verify

Phase 1: Explore

# First connection: gather state — tables, permissions, process info
r = ssh_run("mysql -h db -u test -ptest123 rag -e 'SELECT * FROM permission WHERE ...'")
r = ssh_run("ps aux | grep ragserver | grep -v grep")
r = ssh_run("grep 'storage_stats' path/to/index.ui")

Phase 2: Check before acting

# Check if a DB record exists before INSERTING
r = ssh_run("mysql -h db -u test -ptest123 rag -e \"SELECT id FROM permission WHERE path LIKE '%my_endpoint%'\"")
if 'my_perm' not in r['out']:
    # Only now INSERT
    ssh_run("mysql -h db -u test -ptest123 rag -e \"INSERT INTO permission ...\"")

Pitfall: one script per connection, not one SSH session

Don't try to reuse one paramiko SSHClient for all steps in a single long-lived script. Each exec_command() creates an independent shell — environment variables, cd commands, and source calls don't persist between them. Use separate short scripts with fresh connections per phase:

# WRONG — no state persistence between commands
ssh = paramiko.SSHClient(...)
stdout, _, _ = ssh.exec_command("cd /app && source py3/bin/activate")
stdout, _, _ = ssh.exec_command("python -B app.py")  # still in home dir, no venv

# RIGHT — use && chaining for multi-command sequences
ssh_run("cd /app && source py3/bin/activate && python -B app.py")

# Or use separate scripts with full absolute paths in each

Pitfall: process ownership blocks restart

The SSH user may not own the target process. Always check process ownership before trying to kill:

r = ssh_run("ps aux | grep app_name | grep -v grep")
# Check if Owned_by matches SSH_USER
# If not, pkill will fail with "Operation not permitted"

When process ownership mismatch exists:

• Git pull, DB queries, redis, and curl work fine (file/DB access)
• pkill and systemctl restart will fail
• You need either: SSH as the owning user, su - <owner>, or sudo

Testing with Python execute_code

For complex multi-step tests, write the test script to a file and execute via SSH in a single execute_code call:

# Write test script locally, then run via SSH
# Use subprocess to execute python3 -c '...' with paramiko
# Keep each SSH session focused (single purpose)
# Use separate connections for each phase to avoid timeout

Cookie Management

• Login creates session cookie stored in a file (-c /tmp/cookie.txt)
• Subsequent requests use that cookie (-b /tmp/cookie.txt)
• Each test phase should get a fresh cookie by re-logging in
• Cookie files persist on the remote server between SSH commands