---
name: remote-app-testing
category: devops
description: SSH-based remote application testing workflow for deployed Hermes applications
trigger: When asked to test, verify, or diagnose issues on a remote deployed Hermes application
---

# Remote Application Testing Workflow

## Sage Test Servers

| Server | SSH | Sage Path | DB | Notes |
|--------|-----|-----------|----|----|
| tokentest | `apitest@120.48.168.15` (tokentest.opencomputing.cn) | `/d/apitest/sage --port 9180`, pkgs/ | `mysql -u test -ptest123 sage` | SSH限流, Sage重启后等30s重连 |

Sage部署: `git push` → SSH `git pull` in pkgs/ → `load_path.py` → `pkill sage.py` → restart。

## Overview

Test deployed Hermes applications on remote servers via SSH without direct console access. Uses Python + paramiko for SSH automation and curl for HTTP testing.

## Prerequisites

```python
import paramiko

SSH_HOST = "crm.opencomputing.cn"
SSH_USER = "ymq"
SSH_PASS = "password"

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(SSH_HOST, port=22, username=SSH_USER, password=SSH_PASS, timeout=15)
```

## SSH Command Pattern

```python
def run(ssh, cmd, timeout=15):
    stdin, stdout, stderr = ssh.exec_command(cmd, timeout=timeout)
    out = stdout.read().decode(errors="replace")
    err = stderr.read().decode(errors="replace")
    rc = stdout.channel.recv_exit_status()
    return out, err, rc
```

**Critical:** Always use `|||` as delimiter between body and HTTP status code in curl tests:
```python
curl_cmd = "curl -s -w '|||%{http_code}' -b /tmp/cookie.txt http://localhost:8080/path"
out, _, _ = run(ssh, curl_cmd, timeout=15)
parts = out.split("|||")
body = "|||".join(parts[:-1])
code = int(parts[-1].strip())
```

**Why `|||` not `\n`:** curl's `%{http_code}` can be concatenated to the body without newline in some cases. Using `|||` as delimiter ensures reliable separation.

**Important curl syntax:** When embedding curl commands in triple-quoted Python strings, use `%{http_code}` NOT `%%{http_code}` — the `%` is NOT a Python format specifier inside `run(ssh, cmd)` since cmd is a plain string. However, when using `python3 -c '...'` with single quotes, use `%%{http_code}` to escape for shell.

## Standard Test Phases

### Phase 1: Service Health
```bash
ps aux | grep app_name | grep -v grep
curl -s -w '|||%{http_code}' http://localhost:8080/
tail -20 /tmp/app.log
```

### Phase 2: Public Pages (no auth required)
Test pages that should be accessible without login:
- Login page: `/main/rbac/user/login.ui`
- Register page: `/main/rbac/user/register.ui`
Expected: HTTP 200

### Phase 3: Authentication
```bash
curl -s -c /tmp/test_cookie.txt -X POST \
  http://localhost:8080/main/rbac/user/up_login.dspy \
  -d 'username=admin&password=pass123'
```
Check response contains login success widget (Message widget).

### Phase 4: Authenticated Access
```bash
# First login to get cookie
curl -s -c /tmp/test_cookie.txt -X POST \
  http://localhost:8080/main/rbac/user/up_login.dspy \
  -d 'username=admin&password=pass123' > /dev/null

# Then test protected pages with cookie
curl -s -b /tmp/test_cookie.txt http://localhost:8080/main/module/page.ui
```
Expected: HTTP 200 for authorized pages

### Phase 5: Access Control (negative tests)
```bash
# Without cookie, protected pages should return 401
curl -s -o /dev/null -w '|||%{http_code}' \
  http://localhost:8080/main/module/protected.dspy
```
Expected: HTTP 401

### Phase 6: CRUD Operations
```bash
# Create
curl -s -b /tmp/cookie.txt -X POST \
  http://localhost:8080/main/module/entity/add_entity.dspy \
  -d 'field1=value1&field2=value2'

# Read
curl -s -b /tmp/cookie.txt \
  http://localhost:8080/main/module/entity/get_entity.dspy

# Update
curl -s -b /tmp/cookie.txt -X POST \
  http://localhost:8080/main/module/entity/update_entity.dspy \
  -d 'id=xxx&field1=newvalue'

# Delete
curl -s -b /tmp/cookie.txt -X POST \
  http://localhost:8080/main/module/entity/delete_entity.dspy \
  -d 'id=xxx'
```

### Phase 7: Database State
```bash
mysql -u user -ppass -h db dbname -e "SELECT COUNT(*) FROM table"
mysql -u user -ppass -h db dbname -e "SELECT * FROM table LIMIT 5"
```

### Phase 8: Server Logs
```bash
tail -50 /tmp/app.log
```

## Service Restart Pattern

### Pitfall: kill/pkill kills SSH session

On remote servers where the SSH session runs under the same user as the target processes, `kill -9`, `pkill -f`, or `sudo pkill` will often terminate the SSH connection itself (exit code 255). The restart commands that follow never execute.

**Workaround A — detached background script**:

```bash
# Write kill script, then run via nohup so it survives SSH disconnect
ssh user@host 'nohup bash -c "pkill -9 -f app.py; sleep 3" > /tmp/kill.log 2>&1 &'
sleep 6
ssh user@host 'pgrep -f app.py | wc -l'  # reconnect and verify
```

**Workaround B — kill by PID with SIGTERM first**:

```bash
PIDS=$(pgrep -f "app.py")
kill $PIDS 2>/dev/null      # SIGTERM — less likely to kill SSH
sleep 3
# Only SIGKILL if necessary
kill -9 $(pgrep -f "app.py") 2>/dev/null
```

### Pitfall: code changes must be local-first

**NEVER modify source files directly on the server.** User explicitly corrected this. Always:

1. Modify locally in `~/repos/<module>/`
2. `git add` + `git commit` + `git push`
3. SSH to server: `cd pkgs/<module> && git pull origin main`
4. If Python package: `pip install -e .` (editable) or copy to site-packages
5. Restart service

Server-side file edits are lost on next `git pull` and cause drift. Only exception: temporary debug logging (must revert before commit).

### Standard restart (paramiko)

```python
import time

# Kill
run(ssh, "pkill -f app_name.py 2>/dev/null || true")
time.sleep(2)

# Start
transport = ssh.get_transport()
chan = transport.open_session(timeout=10)
chan.exec_command("cd /path/to/app && source py3/bin/activate && export PYTHONPATH=/path/to/app && nohup python app/app_name.py --port 8080 > /tmp/app.log 2>&1 &")
time.sleep(3)
chan.close()

# Verify
for i in range(10):
    time.sleep(2)
    out, _, _ = run(ssh, "ps aux | grep app_name | grep -v grep")
    if out.strip():
        break
```

## Stress / Concurrency Testing

Test at 3 levels: 30, 60, 100 concurrency. Each test runs for 60 seconds. Use an async Python script with `aiohttp` + `asyncio.Semaphore`.

### Resource monitoring

Sample in parallel with requests:
- **CPU**: `/proc/stat` every 1s
- **Network**: `/proc/net/dev` every 2s
- **Disk IO**: `/proc/diskstats` every 2s

### Metrics collected

- Total requests, success/fail counts, failure type breakdown (timeout vs HTTP errors)
- TTFB and total time: count, min, p50, p75, p90, p95, p99, max, avg
- CPU: min, max, avg, p50, p95
- Network: rx/tx KB/s avg and max
- Disk: read/write IOPS avg and max

### Script workflow

1. Write stress script locally with all monitoring built in
2. `scp` to server
3. Run sequentially: 30 → 60 → 100 concurrency
4. Save JSON report per test to `/tmp/stress_c{N}_v2.json`
5. Compare across concurrency levels

### Profiling before optimization

Before optimizing, instrument the hot path with timing probes to identify the actual bottleneck. Use a lightweight timing module that monkey-patches `json.loads/dumps` and provides `@timed` decorators for async functions. Collect per-request breakdown, then revert all instrumentation. Never profile in production — only in test environment with user consent.

Reference template: `references/llm_stress_template.py` — minimal async stress test script without resource monitors. Full version with CPU/network/disk monitoring at `/d/ymq/llm_stress_v2.py`.

## Common Issues to Check

1. **Service not running** → Check process list, restart
2. **401 Unauthorized on public pages** → Check anonymous/any role permissions in DB
3. **404 Not Found** → Check wwwroot symlinks exist: `ls -la wwwroot/`
4. **500 Internal Server Error** → Check server log tail, look for Python exceptions
5. **Path resolution errors** → Check ahserver url2file.py bug (prefix not stripped)
6. **Permission not enforced** → Check rbac userperm.py wildcard matching bug
7. **True/false syntax errors** → .dspy files using JavaScript `true` instead of Python `True`

## File Upload Pattern

```python
sftp = ssh.open_sftp()
sftp.put("/local/path/file.py", "/remote/path/file.py")
sftp.close()
```

## Important URL Mapping Rule

In Hermes apps, URL paths map to filesystem as:
- Config: `["wwwroot", "/main"]` means `wwwroot/` serves at `/main/`
- URL `/main/rbac/user/login.ui` → Filesystem `wwwroot/rbac/user/login.ui`
- The `/main` prefix is STRIPPED before filesystem lookup

If symlinks exist in wwwroot pointing to module wwwroot directories:
- `wwwroot/rbac` → `pkgs/rbac/wwwroot`
- `wwwroot/customer_management` → `pkgs/customer_management/wwwroot`

## Progressive Exploration Pattern

For multi-step deploy-test tasks, use a **progressively deepening** approach: explore first, then act. This avoids blindly executing steps that may already be satisfied.

### Pattern: Sequential paramiko scripts with fresh connections

Rather than one monolithic script, write a series of small scripts, each focused on one phase, run sequentially:

```
script1.py   →  explore (what exists? what state?)
script2.py   →  check specific conditions
script3.py   →  act based on findings
script4.py   →  verify
```

### Phase 1: Explore

```python
# First connection: gather state — tables, permissions, process info
r = ssh_run("mysql -h db -u test -ptest123 rag -e 'SELECT * FROM permission WHERE ...'")
r = ssh_run("ps aux | grep ragserver | grep -v grep")
r = ssh_run("grep 'storage_stats' path/to/index.ui")
```

### Phase 2: Check before acting

```python
# Check if a DB record exists before INSERTING
r = ssh_run("mysql -h db -u test -ptest123 rag -e \"SELECT id FROM permission WHERE path LIKE '%my_endpoint%'\"")
if 'my_perm' not in r['out']:
    # Only now INSERT
    ssh_run("mysql -h db -u test -ptest123 rag -e \"INSERT INTO permission ...\"")
```

### Pitfall: one script per connection, not one SSH session

**Don't** try to reuse one paramiko SSHClient for all steps in a single long-lived script. Each `exec_command()` creates an independent shell — environment variables, `cd` commands, and `source` calls don't persist between them. Use **separate short scripts with fresh connections per phase**:

```python
# WRONG — no state persistence between commands
ssh = paramiko.SSHClient(...)
stdout, _, _ = ssh.exec_command("cd /app && source py3/bin/activate")
stdout, _, _ = ssh.exec_command("python -B app.py")  # still in home dir, no venv

# RIGHT — use && chaining for multi-command sequences
ssh_run("cd /app && source py3/bin/activate && python -B app.py")

# Or use separate scripts with full absolute paths in each
```

### Pitfall: process ownership blocks restart

The SSH user may not own the target process. Always check process ownership before trying to kill:

```python
r = ssh_run("ps aux | grep app_name | grep -v grep")
# Check if Owned_by matches SSH_USER
# If not, pkill will fail with "Operation not permitted"
```

When process ownership mismatch exists:
- Git pull, DB queries, redis, and curl work fine (file/DB access)
- `pkill` and `systemctl restart` will fail
- You need either: SSH as the owning user, `su - <owner>`, or sudo

## Testing with Python execute_code

For complex multi-step tests, write the test script to a file and execute via SSH in a single execute_code call:

```python
# Write test script locally, then run via SSH
# Use subprocess to execute python3 -c '...' with paramiko
# Keep each SSH session focused (single purpose)
# Use separate connections for each phase to avoid timeout
```

## Cookie Management

- Login creates session cookie stored in a file (`-c /tmp/cookie.txt`)
- Subsequent requests use that cookie (`-b /tmp/cookie.txt`)
- Each test phase should get a fresh cookie by re-logging in
- Cookie files persist on the remote server between SSH commands