---
name: ahserver-hot-reload
version: 1.0.0
description: ahserver file-based hot-reload system for config, i18n, and module caches (multi-process safe)
trigger_conditions:
  - User asks how to enable/configure ahserver hot-reload
  - User needs to clear module caches without restarting
  - User is debugging stale config/i18n/cache in multi-worker deployment
  - User asks about /__hot_reload__ endpoint
  - User needs to distribute cache invalidation across multiple workers
---

# ahserver Hot-Reload System

File-based hot-reload for ahserver, watching config.json and i18n files. Multi-process safe (each worker independently checks file mtimes).

Code: `ahserver/ahserver/hotreload.py`, `ahserver/ahserver/webapp.py`

## Enable Hot-Reload

Add to `conf/config.json`:

```json
{
    "hot_reload": true
}
```

Or with custom interval:

```json
{
    "hot_reload": {
        "enabled": true,
        "interval": 2
    }
}
```

Restart ahserver after enabling.

## What Gets Auto-Reloaded

| Trigger | Config Singleton | Module Caches (hot_reload event) |
|---------|-----------------|----------------------------------|
| `conf/config.json` mtime change | ✓ Cleared (next getConfig reloads) | ✗ NOT dispatched |
| `i18n/*/msg.txt` mtime change | ✗ | ✓ Dispatched |
| Signal file mtime change (cross-worker) | ✗ | ✓ Dispatched |
| `GET /__hot_reload__` endpoint | ✗ | ✓ Dispatched (also writes signal file) |

**Key design**: config.json changes only refresh the JsonConfig singleton. Module caches are NOT cleared because config changes rarely affect cached module data. Only i18n changes, signal file updates, or explicit HTTP calls trigger cache clearing.

## Manual Cache Invalidation

HTTP endpoint to clear all module caches without file changes:

```bash
curl http://localhost:PORT/__hot_reload__
```

Returns:
```json
{
    "status": "ok",
    "message": "Signal sent to all workers, current worker dispatched hot_reload",
    "timestamp": 1738416000.0
}
```

## Module Caches Cleared (via EventDispatcher)

Each module implements `on_hot_reload(data=None)` on its cache-holding class/instance, bound in `load_XXX()`:

| Module | Cache | Clear Method | Bound On |
|--------|-------|--------------|----------|
| rbac | User permissions (LRUCache), role-permissions (dict→None) | `UserPermissions.on_hot_reload()` | Instance method (stored on ServerEnv) |
| pricing | Pricing data per org (class-level dict) | `PricingProgram.on_hot_reload()` | @staticmethod (class-level) |
| uapi | API users, API definitions, API keys (3 dicts) | `UAPIData.on_hot_reload()` | Instance method (stored on ServerEnv) |
| llmage | LLM API/uapiio cache (module-level dicts) | `_on_hot_reload()` wrapper | Module-level function (module keeps it alive) |

### ⚠️ CRITICAL: rbac is NOT a singleton

`UserPermissions` does NOT use `@SingletonDecorator`. The actual instance is created once in `rbac/load_rbac()` and stored on `ServerEnv().userpermissions`. Creating `UserPermissions()` anywhere else gives you a **new empty instance** with empty caches — clearing it does nothing.

**Wrong**:
```python
from rbac.userperm import UserPermissions
up = UserPermissions()  # ← NEW empty instance, not the real one
up.ur_caches.clear()    # ← clears nothing useful
```

**Correct**:
```python
from ahserver.serverenv import ServerEnv
g = ServerEnv()
up = g.userpermissions  # ← the actual instance with real caches
up.ur_caches.clear()
up.invalidate_rp_cache()
```

## Multi-Worker Deployment

ahserver runs multiple workers (via `reuse_port=True`). Each worker has independent Python memory space.

### Problem

`GET /__hot_reload__` only clears cache in the worker that receives the request. Other workers still have stale cache.

### Solutions (choose based on deployment)

#### Solution 1: Shell Loop (simplest, reuse_port multi-port)

Each worker listens on different port:

```bash
#!/bin/bash
# hot_reload_all.sh
PORTS=(8000 8001 8002 8003)
for port in "${PORTS[@]}"; do
    curl -s "http://127.0.0.1:$port/__hot_reload__" &
done
wait
echo "Done"
```

**When to use**: reuse_port mode with explicit port assignment.

Ready-to-use script: `scripts/hot_reload_all.sh` — pass ports as args or defaults to 8000-8003.

#### Solution 2: nginx mirror directive (automatic replication)

```nginx
upstream worker_0 { server 127.0.0.1:8000; }
upstream worker_1 { server 127.0.0.1:8001; }
upstream worker_2 { server 127.0.0.1:8002; }
upstream worker_3 { server 127.0.0.1:8003; }

server {
    listen 80;

    location /__hot_reload__ {
        mirror /__hot_reload_mirror_1__;
        mirror /__hot_reload_mirror_2__;
        mirror /__hot_reload_mirror_3__;
        
        proxy_pass http://worker_0;
    }

    location = /__hot_reload_mirror_1__ {
        internal;
        proxy_pass http://worker_1/__hot_reload__;
    }
    location = /__hot_reload_mirror_2__ {
        internal;
        proxy_pass http://worker_2/__hot_reload__;
    }
    location = /__hot_reload_mirror_3__ {
        internal;
        proxy_pass http://worker_3/__hot_reload__;
    }
}
```

**When to use**: nginx as load balancer, want single curl to hit all workers.

**Pitfall**: nginx mirror is fire-and-forget — client doesn't see mirror responses. If a worker fails, you won't know from the main response.

#### Solution 3: File Signal (IMPLEMENTED — production default)

Already implemented in `hotreload.py` (commit 42eff6c). No code changes needed.

**How it works**:
1. `GET /__hot_reload__` hits any worker via nginx
2. That worker writes timestamp to `/tmp/.sage_cache_invalidate` and dispatches `hot_reload` immediately
3. All other workers' `HotReloader._check_signal_file()` detects mtime change within `interval` seconds (default 2s)
4. All workers dispatch `hot_reload` event → each module's bound handler clears its own cache
5. Each worker clears its own caches independently

**Single curl is sufficient** — no shell loop or nginx config needed:
```bash
curl http://localhost:PORT/__hot_reload__
```

Response only shows the worker that received the request, but ALL workers will clear caches within ~2s.

## Logs

**INFO level** (default):
```
[hot_reload] started, interval=2s
[hot_reload] reloaded: ['config', 'i18n']
[hot_reload] reloaded: ['signal']
[hot_reload] stopped
```

**DEBUG level** (set `logger.levelname: "debug"` in config.json):
```
[hot_reload] config_path=/path/to/conf/config.json
[hot_reload] watching 2 i18n paths
[hot_reload] initial mtime for /path/to/file: 1717257600.0
[hot_reload] changed: /path/to/file (mtime 1717257600.0 -> 1717257700.0)
[hot_reload] signal file mtime: 1717257700.0, last: 0
[hot_reload] signal file changed, triggering reload
[hot_reload] config changed: ['/path/to/conf/config.json']
[hot_reload] clearing JsonConfig singleton
[hot_reload] clearing MiniI18N singleton
[hot_reload] cleared ServerEnv.myi18n
[hot_reload] config-only change, skipping cache clear dispatch
[hot_reload] dispatching hot_reload event (non-config changes detected)
[hot_reload] HTTP endpoint triggered, writing signal to /tmp/.sage_cache_invalidate
[hot_reload] HTTP endpoint: dispatching hot_reload event
```

**Module handler logs** (DEBUG level):
```
[uapi] on_hot_reload called, clearing caches (data={...})
[rbac] on_hot_reload called, clearing caches (data={...})
[pricing] on_hot_reload called, clearing pricing_data (data={...})
[llmage] on_hot_reload called, invalidating uapi cache (data={...})
```

**Troubleshooting**: If hot_reload isn't triggering cache clears, enable DEBUG logging and check:
1. File mtime changes are detected (look for `changed:` log)
2. Whether it's config-only (look for `config-only change, skipping` vs `dispatching`)
3. Whether module handlers are called (look for `on_hot_reload called` logs)
4. If handler logs missing, check the module's `load_XXX()` bind call — WeakCallback may have lost the reference

## Limitations

1. **No Python code hot-reload** — Only config/i18n/cache. Code changes require restart.
2. **File mtime resolution** — On some filesystems (NFS, Docker volumes), mtime may not update immediately.
3. **Signal file latency** — Multi-worker cache clear has ~2s delay (configurable via `interval`). Not instant like Redis Pub/Sub would be.

## Comparison with Redis Pub/Sub cache_sync

| Feature | hot-reload (this) | cache_sync (Redis) |
|---------|-------------------|-------------------|
| Trigger | File change / HTTP | Database event |
| Infrastructure | None | Redis |
| Latency | 2s (polling) | Instant |
| Status | **Production ready** | Reverted (session loss bug) |
| Use case | Dev/testing, manual invalidation | Production auto-sync |

See `sage-cache-sync` skill for Redis Pub/Sub approach (currently reverted).

## EventDispatcher Architecture (Implemented)

Uses `appPublic.event_dispatcher.EventDispatcher` (NOT `eventpy`) — implements WeakCallback with weakref for automatic cleanup. See `references/event-dispatcher-api.md` for full API reference.

### Key API

```python
class EventDispatcher:
    def bind(self, event_name: str, func: Callable)      # register handler (WeakCallback)
    def unbind(self, event_name: str, func: Callable)     # unregister
    async def dispatch(self, event_name: str, data=None)  # fire event, await all handlers
```

Handlers receive `data` argument (the reloaded dict or custom payload). Both sync and async handlers are supported.

### Lifecycle

```
webserver() in webapp.py:
  1. se.event_dispatcher = EventDispatcher()   ← BEFORE init_func()
  2. init_func() → load_rbac/pricing/uapi/llmage → each binds 'hot_reload'
  3. ConfiguredServer → server.run()

Runtime triggers → dispatch('hot_reload'):
  - GET /__hot_reload__ → writes signal file + immediate dispatch
  - signal file mtime change (other workers) → dispatch
  - i18n file mtime change → dispatch

Config.json mtime change → reloads JsonConfig singleton ONLY, does NOT dispatch hot_reload.
```

### Adding a New Module's Cache Clear

In your module's class, add `on_hot_reload`:

```python
class MyModule:
    def __init__(self):
        self.cache = {}
    
    def on_hot_reload(self, data=None):
        self.cache.clear()
```

In `load_mymodule()`:

```python
def load_mymodule():
    env = ServerEnv()
    env.mymodule = MyModule()
    # Guard for non-web contexts (scripts, tests)
    # CRITICAL: use getattr + None check, NOT hasattr
    # hasattr only checks attribute existence, but event_dispatcher
    # can exist as None when running standalone (e.g. backend_accounting.py)
    if getattr(env, 'event_dispatcher', None) is not None:
        env.event_dispatcher.bind('hot_reload', env.mymodule.on_hot_reload)
```

### ⚠️ CRITICAL: WeakCallback Pitfalls

EventDispatcher uses `weakref.ref` for functions and `weakref.WeakMethod` for instance methods. If the handler's target gets garbage-collected, the binding silently disappears.

**Wrong — lambda gets GC'd immediately:**
```python
env.event_dispatcher.bind('hot_reload', lambda data: cache.clear())
# lambda has no strong reference → GC'd → binding lost
```

**Wrong — local function gets GC'd:**
```python
def load_mymodule():
    async def clear(data):      # local function
        cache.clear()
    env.event_dispatcher.bind('hot_reload', clear)
    # clear() is local → GC'd after load_mymodule() returns → binding lost
```

**Correct patterns:**

| Pattern | Why it works |
|---------|-------------|
| Instance method on object stored on `ServerEnv` | `ServerEnv` holds strong ref to instance → WeakMethod stays valid |
| `@staticmethod` on a class | Class is never GC'd → ref stays valid |
| Module-level function | Module stays loaded → ref stays valid |

### Signature Requirement

All handlers receive `data` as argument. If wrapping an existing function that doesn't accept args:

```python
# llmage's invalidate_uapi_cache() takes optional upappid/apiname
# dispatcher calls with data=dict → need wrapper
def _on_hot_reload(data=None):
    invalidate_uapi_cache()

env.event_dispatcher.bind('hot_reload', _on_hot_reload)
```

## Pitfalls

### rbac UserPermissions is not a singleton

`UserPermissions()` creates a new empty instance. Always use `ServerEnv().userpermissions` to get the actual instance with real caches. See:
- `references/rbac-non-singleton-pitfall.md` — why this happens and how to avoid it
- `references/rbac-event-handler-bug.md` — unfixed bug in rbac/init.py event handlers (same root cause)

### hasattr vs getattr for event_dispatcher — use getattr with None check

`hasattr(env, 'event_dispatcher')` only checks attribute existence. In standalone scripts (e.g., `backend_accounting.py`), `event_dispatcher` exists on `ServerEnv` but its value is `None`. This causes `AttributeError: 'NoneType' object has no attribute 'bind'`.

**Wrong:**
```python
if hasattr(env, 'event_dispatcher'):
    env.event_dispatcher.bind('hot_reload', handler)  # ← crashes if event_dispatcher is None
```

**Correct:**
```python
if getattr(env, 'event_dispatcher', None) is not None:
    env.event_dispatcher.bind('hot_reload', handler)
```

### Debug log noise in periodic tasks

The hot_reload task runs every N seconds and checks multiple file mtimes. Debug logs that fire unconditionally on every check cycle flood the log file and obscure real events.

**Wrong — logs every 2s even when nothing changes:**
```python
def _check_signal_file(self):
    mtime = os.path.getmtime(SIGNAL_FILE)
    debug(f'[hot_reload] signal file mtime: {mtime}, last: {self._last_signal_mtime}')  # ← noise
    if mtime > self._last_signal_mtime:
        ...
```

**Correct — only log when state actually changes:**
```python
def _check_signal_file(self):
    mtime = os.path.getmtime(SIGNAL_FILE)
    if mtime > self._last_signal_mtime:
        self._last_signal_mtime = mtime
        debug(f'[hot_reload] signal file changed, mtime: {mtime}')  # ← only on change
        return True
```

**Same applies to OSError on missing files** — the signal file may not exist for hours. Don't log "not found" on every check; silently pass.

**General rule for periodic task debug logging**: Gate log statements behind the condition that makes them interesting. "Checked X" is noise; "X changed from A to B" is signal.

### Config.json must be valid JSON

Hot-reload clears JsonConfig singleton, next `getConfig()` reloads from disk. If config.json has syntax error, server will crash on next config access.

**Fix**: Validate config.json before saving.

### aiohttp cleanup_ctx vs on_cleanup

`app.cleanup_ctx.append()` requires an **async context manager** (must `yield`). Plain `async def` functions that don't yield cause `AttributeError: 'coroutine' object has no attribute '__aiter__'`.

| API | Accepts | Use for |
|-----|---------|---------|
| `app.cleanup_ctx.append()` | `async def f(app): ... yield ...` (async context manager) | Need setup + teardown in one function |
| `app.on_cleanup.append()` | `async def f(app): ...` (plain coroutine) | Teardown-only cleanup (e.g. cancel task) |

**Bug in hot_reload**: `_hot_reload_cleanup` was a plain async def added to `cleanup_ctx`. Fixed by switching to `on_cleanup.append()`.

**Symptom**:
```
AttributeError: 'coroutine' object has no attribute '__aiter__'. Did you mean: '__dir__'?
sys:1: RuntimeWarning: coroutine '_hot_reload_cleanup' was never awaited
```

### i18n file path detection

`get_i18n_paths()` scans `i18n/*/msg.txt`. If you add a new language directory after hot-reload starts, it won't be watched until restart.

**Fix**: Restart after adding new language.

### Signal file detection

Signal file at `/tmp/.sage_cache_invalidate` — all workers detect mtime change and dispatch `hot_reload` event.

### Module import errors (no longer applies)

Previously `invalidate_all_caches()` imported modules directly. Now uses EventDispatcher — modules self-register. If a module doesn't bind, its cache won't be cleared (check its `load_XXX()` for the bind call).

### Git force-commit needed to resync truncated files

When a file is truncated in the server's working copy but the local repo already has the correct version at HEAD, `git checkout HEAD` reports no change and `git pull` says "up to date". The server never gets the fix.

**Symptom**: Server returns 500 because a function is missing from a truncated file, but `git pull` on server shows nothing to update.

**Root cause**: The file was modified locally (truncated), committed, then restored via `git checkout HEAD`. Now local and remote HEAD are identical — the correct file is in git history. But the server's working copy still has the old truncated version.

**Fix**: Force a commit that changes the file, even trivially:
```bash
# Add a comment or whitespace to create a diff
echo "# Force re-sync" >> path/to/file.py
git add path/to/file.py
git commit -m "force: re-sync <file> (ensure full version)"
git push
```

Then server `git pull` will pull the new commit and overwrite the truncated file.