pipeline_core/skills_library/all/ahserver-hot-reload/SKILL.md

name	version	description	trigger_conditions
ahserver-hot-reload	1.0.0	ahserver file-based hot-reload system for config, i18n, and module caches (multi-process safe)	User asks how to enable/configure ahserver hot-reload User needs to clear module caches without restarting User is debugging stale config/i18n/cache in multi-worker deployment User asks about /__hot_reload__ endpoint User needs to distribute cache invalidation across multiple workers

ahserver Hot-Reload System

File-based hot-reload for ahserver, watching config.json and i18n files. Multi-process safe (each worker independently checks file mtimes).

Code: ahserver/ahserver/hotreload.py, ahserver/ahserver/webapp.py

Enable Hot-Reload

Add to conf/config.json:

{
    "hot_reload": true
}

Or with custom interval:

{
    "hot_reload": {
        "enabled": true,
        "interval": 2
    }
}

Restart ahserver after enabling.

What Gets Auto-Reloaded

Trigger	Config Singleton	Module Caches (hot_reload event)
conf/config.json mtime change	✓ Cleared (next getConfig reloads)	✗ NOT dispatched
i18n/*/msg.txt mtime change	✗	✓ Dispatched
Signal file mtime change (cross-worker)	✗	✓ Dispatched
GET /__hot_reload__ endpoint	✗	✓ Dispatched (also writes signal file)

Key design: config.json changes only refresh the JsonConfig singleton. Module caches are NOT cleared because config changes rarely affect cached module data. Only i18n changes, signal file updates, or explicit HTTP calls trigger cache clearing.

Manual Cache Invalidation

HTTP endpoint to clear all module caches without file changes:

curl http://localhost:PORT/__hot_reload__

Returns:

{
    "status": "ok",
    "message": "Signal sent to all workers, current worker dispatched hot_reload",
    "timestamp": 1738416000.0
}

Module Caches Cleared (via EventDispatcher)

Each module implements on_hot_reload(data=None) on its cache-holding class/instance, bound in load_XXX():

Module	Cache	Clear Method	Bound On
rbac	User permissions (LRUCache), role-permissions (dict→None)	UserPermissions.on_hot_reload()	Instance method (stored on ServerEnv)
pricing	Pricing data per org (class-level dict)	PricingProgram.on_hot_reload()	@staticmethod (class-level)
uapi	API users, API definitions, API keys (3 dicts)	UAPIData.on_hot_reload()	Instance method (stored on ServerEnv)
llmage	LLM API/uapiio cache (module-level dicts)	_on_hot_reload() wrapper	Module-level function (module keeps it alive)

⚠️ CRITICAL: rbac is NOT a singleton

UserPermissions does NOT use @SingletonDecorator. The actual instance is created once in rbac/load_rbac() and stored on ServerEnv().userpermissions. Creating UserPermissions() anywhere else gives you a new empty instance with empty caches — clearing it does nothing.

Wrong:

from rbac.userperm import UserPermissions
up = UserPermissions()  # ← NEW empty instance, not the real one
up.ur_caches.clear()    # ← clears nothing useful

Correct:

from ahserver.serverenv import ServerEnv
g = ServerEnv()
up = g.userpermissions  # ← the actual instance with real caches
up.ur_caches.clear()
up.invalidate_rp_cache()

Multi-Worker Deployment

ahserver runs multiple workers (via reuse_port=True). Each worker has independent Python memory space.

Problem

GET /__hot_reload__ only clears cache in the worker that receives the request. Other workers still have stale cache.

Solutions (choose based on deployment)

Solution 1: Shell Loop (simplest, reuse_port multi-port)

Each worker listens on different port:

#!/bin/bash
# hot_reload_all.sh
PORTS=(8000 8001 8002 8003)
for port in "${PORTS[@]}"; do
    curl -s "http://127.0.0.1:$port/__hot_reload__" &
done
wait
echo "Done"

When to use: reuse_port mode with explicit port assignment.

Ready-to-use script: scripts/hot_reload_all.sh — pass ports as args or defaults to 8000-8003.

Solution 2: nginx mirror directive (automatic replication)

upstream worker_0 { server 127.0.0.1:8000; }
upstream worker_1 { server 127.0.0.1:8001; }
upstream worker_2 { server 127.0.0.1:8002; }
upstream worker_3 { server 127.0.0.1:8003; }

server {
    listen 80;

    location /__hot_reload__ {
        mirror /__hot_reload_mirror_1__;
        mirror /__hot_reload_mirror_2__;
        mirror /__hot_reload_mirror_3__;
        
        proxy_pass http://worker_0;
    }

    location = /__hot_reload_mirror_1__ {
        internal;
        proxy_pass http://worker_1/__hot_reload__;
    }
    location = /__hot_reload_mirror_2__ {
        internal;
        proxy_pass http://worker_2/__hot_reload__;
    }
    location = /__hot_reload_mirror_3__ {
        internal;
        proxy_pass http://worker_3/__hot_reload__;
    }
}

When to use: nginx as load balancer, want single curl to hit all workers.

Pitfall: nginx mirror is fire-and-forget — client doesn't see mirror responses. If a worker fails, you won't know from the main response.

Solution 3: File Signal (IMPLEMENTED — production default)

Already implemented in hotreload.py (commit 42eff6c). No code changes needed.

How it works:

1. GET /__hot_reload__ hits any worker via nginx
2. That worker writes timestamp to /tmp/.sage_cache_invalidate and dispatches hot_reload immediately
3. All other workers' HotReloader._check_signal_file() detects mtime change within interval seconds (default 2s)
4. All workers dispatch hot_reload event → each module's bound handler clears its own cache
5. Each worker clears its own caches independently

Single curl is sufficient — no shell loop or nginx config needed:

curl http://localhost:PORT/__hot_reload__

Response only shows the worker that received the request, but ALL workers will clear caches within ~2s.

Logs

INFO level (default):

[hot_reload] started, interval=2s
[hot_reload] reloaded: ['config', 'i18n']
[hot_reload] reloaded: ['signal']
[hot_reload] stopped

DEBUG level (set logger.levelname: "debug" in config.json):

[hot_reload] config_path=/path/to/conf/config.json
[hot_reload] watching 2 i18n paths
[hot_reload] initial mtime for /path/to/file: 1717257600.0
[hot_reload] changed: /path/to/file (mtime 1717257600.0 -> 1717257700.0)
[hot_reload] signal file mtime: 1717257700.0, last: 0
[hot_reload] signal file changed, triggering reload
[hot_reload] config changed: ['/path/to/conf/config.json']
[hot_reload] clearing JsonConfig singleton
[hot_reload] clearing MiniI18N singleton
[hot_reload] cleared ServerEnv.myi18n
[hot_reload] config-only change, skipping cache clear dispatch
[hot_reload] dispatching hot_reload event (non-config changes detected)
[hot_reload] HTTP endpoint triggered, writing signal to /tmp/.sage_cache_invalidate
[hot_reload] HTTP endpoint: dispatching hot_reload event

Module handler logs (DEBUG level):

[uapi] on_hot_reload called, clearing caches (data={...})
[rbac] on_hot_reload called, clearing caches (data={...})
[pricing] on_hot_reload called, clearing pricing_data (data={...})
[llmage] on_hot_reload called, invalidating uapi cache (data={...})

Troubleshooting: If hot_reload isn't triggering cache clears, enable DEBUG logging and check:

1. File mtime changes are detected (look for changed: log)
2. Whether it's config-only (look for config-only change, skipping vs dispatching)
3. Whether module handlers are called (look for on_hot_reload called logs)
4. If handler logs missing, check the module's load_XXX() bind call — WeakCallback may have lost the reference

Limitations

1. No Python code hot-reload — Only config/i18n/cache. Code changes require restart.
2. File mtime resolution — On some filesystems (NFS, Docker volumes), mtime may not update immediately.
3. Signal file latency — Multi-worker cache clear has ~2s delay (configurable via interval). Not instant like Redis Pub/Sub would be.

Comparison with Redis Pub/Sub cache_sync

Feature	hot-reload (this)	cache_sync (Redis)
Trigger	File change / HTTP	Database event
Infrastructure	None	Redis
Latency	2s (polling)	Instant
Status	Production ready	Reverted (session loss bug)
Use case	Dev/testing, manual invalidation	Production auto-sync

See sage-cache-sync skill for Redis Pub/Sub approach (currently reverted).

EventDispatcher Architecture (Implemented)

Uses appPublic.event_dispatcher.EventDispatcher (NOT eventpy) — implements WeakCallback with weakref for automatic cleanup. See references/event-dispatcher-api.md for full API reference.

Key API

class EventDispatcher:
    def bind(self, event_name: str, func: Callable)      # register handler (WeakCallback)
    def unbind(self, event_name: str, func: Callable)     # unregister
    async def dispatch(self, event_name: str, data=None)  # fire event, await all handlers

Handlers receive data argument (the reloaded dict or custom payload). Both sync and async handlers are supported.

Lifecycle

webserver() in webapp.py:
  1. se.event_dispatcher = EventDispatcher()   ← BEFORE init_func()
  2. init_func() → load_rbac/pricing/uapi/llmage → each binds 'hot_reload'
  3. ConfiguredServer → server.run()

Runtime triggers → dispatch('hot_reload'):
  - GET /__hot_reload__ → writes signal file + immediate dispatch
  - signal file mtime change (other workers) → dispatch
  - i18n file mtime change → dispatch

Config.json mtime change → reloads JsonConfig singleton ONLY, does NOT dispatch hot_reload.

Adding a New Module's Cache Clear

In your module's class, add on_hot_reload:

class MyModule:
    def __init__(self):
        self.cache = {}
    
    def on_hot_reload(self, data=None):
        self.cache.clear()

In load_mymodule():

def load_mymodule():
    env = ServerEnv()
    env.mymodule = MyModule()
    # Guard for non-web contexts (scripts, tests)
    # CRITICAL: use getattr + None check, NOT hasattr
    # hasattr only checks attribute existence, but event_dispatcher
    # can exist as None when running standalone (e.g. backend_accounting.py)
    if getattr(env, 'event_dispatcher', None) is not None:
        env.event_dispatcher.bind('hot_reload', env.mymodule.on_hot_reload)

⚠️ CRITICAL: WeakCallback Pitfalls

EventDispatcher uses weakref.ref for functions and weakref.WeakMethod for instance methods. If the handler's target gets garbage-collected, the binding silently disappears.

Wrong — lambda gets GC'd immediately:

env.event_dispatcher.bind('hot_reload', lambda data: cache.clear())
# lambda has no strong reference → GC'd → binding lost

Wrong — local function gets GC'd:

def load_mymodule():
    async def clear(data):      # local function
        cache.clear()
    env.event_dispatcher.bind('hot_reload', clear)
    # clear() is local → GC'd after load_mymodule() returns → binding lost

Correct patterns:

Pattern	Why it works
Instance method on object stored on ServerEnv	ServerEnv holds strong ref to instance → WeakMethod stays valid
@staticmethod on a class	Class is never GC'd → ref stays valid
Module-level function	Module stays loaded → ref stays valid

Signature Requirement

All handlers receive data as argument. If wrapping an existing function that doesn't accept args:

# llmage's invalidate_uapi_cache() takes optional upappid/apiname
# dispatcher calls with data=dict → need wrapper
def _on_hot_reload(data=None):
    invalidate_uapi_cache()

env.event_dispatcher.bind('hot_reload', _on_hot_reload)

Pitfalls

rbac UserPermissions is not a singleton

UserPermissions() creates a new empty instance. Always use ServerEnv().userpermissions to get the actual instance with real caches. See:

• references/rbac-non-singleton-pitfall.md — why this happens and how to avoid it
• references/rbac-event-handler-bug.md — unfixed bug in rbac/init.py event handlers (same root cause)

hasattr vs getattr for event_dispatcher — use getattr with None check

hasattr(env, 'event_dispatcher') only checks attribute existence. In standalone scripts (e.g., backend_accounting.py), event_dispatcher exists on ServerEnv but its value is None. This causes AttributeError: 'NoneType' object has no attribute 'bind'.

Wrong:

if hasattr(env, 'event_dispatcher'):
    env.event_dispatcher.bind('hot_reload', handler)  # ← crashes if event_dispatcher is None

Correct:

if getattr(env, 'event_dispatcher', None) is not None:
    env.event_dispatcher.bind('hot_reload', handler)

Debug log noise in periodic tasks

The hot_reload task runs every N seconds and checks multiple file mtimes. Debug logs that fire unconditionally on every check cycle flood the log file and obscure real events.

Wrong — logs every 2s even when nothing changes:

def _check_signal_file(self):
    mtime = os.path.getmtime(SIGNAL_FILE)
    debug(f'[hot_reload] signal file mtime: {mtime}, last: {self._last_signal_mtime}')  # ← noise
    if mtime > self._last_signal_mtime:
        ...

Correct — only log when state actually changes:

def _check_signal_file(self):
    mtime = os.path.getmtime(SIGNAL_FILE)
    if mtime > self._last_signal_mtime:
        self._last_signal_mtime = mtime
        debug(f'[hot_reload] signal file changed, mtime: {mtime}')  # ← only on change
        return True

Same applies to OSError on missing files — the signal file may not exist for hours. Don't log "not found" on every check; silently pass.

General rule for periodic task debug logging: Gate log statements behind the condition that makes them interesting. "Checked X" is noise; "X changed from A to B" is signal.

Config.json must be valid JSON

Hot-reload clears JsonConfig singleton, next getConfig() reloads from disk. If config.json has syntax error, server will crash on next config access.

Fix: Validate config.json before saving.

aiohttp cleanup_ctx vs on_cleanup

app.cleanup_ctx.append() requires an async context manager (must yield). Plain async def functions that don't yield cause AttributeError: 'coroutine' object has no attribute '__aiter__'.

API	Accepts	Use for
app.cleanup_ctx.append()	async def f(app): ... yield ... (async context manager)	Need setup + teardown in one function
app.on_cleanup.append()	async def f(app): ... (plain coroutine)	Teardown-only cleanup (e.g. cancel task)

Bug in hot_reload: _hot_reload_cleanup was a plain async def added to cleanup_ctx. Fixed by switching to on_cleanup.append().

Symptom:

AttributeError: 'coroutine' object has no attribute '__aiter__'. Did you mean: '__dir__'?
sys:1: RuntimeWarning: coroutine '_hot_reload_cleanup' was never awaited

i18n file path detection

get_i18n_paths() scans i18n/*/msg.txt. If you add a new language directory after hot-reload starts, it won't be watched until restart.

Fix: Restart after adding new language.

Signal file detection

Signal file at /tmp/.sage_cache_invalidate — all workers detect mtime change and dispatch hot_reload event.

Module import errors (no longer applies)

Previously invalidate_all_caches() imported modules directly. Now uses EventDispatcher — modules self-register. If a module doesn't bind, its cache won't be cleared (check its load_XXX() for the bind call).

Git force-commit needed to resync truncated files

When a file is truncated in the server's working copy but the local repo already has the correct version at HEAD, git checkout HEAD reports no change and git pull says "up to date". The server never gets the fix.

Symptom: Server returns 500 because a function is missing from a truncated file, but git pull on server shows nothing to update.

Root cause: The file was modified locally (truncated), committed, then restored via git checkout HEAD. Now local and remote HEAD are identical — the correct file is in git history. But the server's working copy still has the old truncated version.

Fix: Force a commit that changes the file, even trivially:

# Add a comment or whitespace to create a diff
echo "# Force re-sync" >> path/to/file.py
git add path/to/file.py
git commit -m "force: re-sync <file> (ensure full version)"
git push

Then server git pull will pull the new commit and overwrite the truncated file.