pipeline_core/skills_library/all/build-script-modularization/SKILL.md

name	title	description
build-script-modularization	Build Script Modularization and Database Separation	Guidelines for separating database setup from application build scripts and implementing modular processing loops

Build Script Modularization and Database Separation

Problem Statement

Build scripts often fail in production environments because they attempt to create databases using root privileges, which is both a security risk and often fails due to permission restrictions. Additionally, repetitive code for processing multiple modules reduces maintainability.

Solution Approach

1. Separate Database Setup

• Create an external setup_database.sh script that handles:
  • Database creation
  • User creation
  • Permission grants
  • This script requires root privileges and should be run separately

2. External Password Encryption

• Create a dedicated encryption script (encrypt_password.py) that:
  • Uses the application's encryption library (e.g., apppublic)
  • Reads existing config files to get encryption keys
  • Updates configuration with encrypted passwords
  • Provides fallback mechanisms for encryption failures

3. Modular Processing Loop

• Define modules as an array: MODULES=("module1" "module2" "module3")
• Use a for loop to process each module uniformly:
  • Clone/install modules
  • Generate database DDL from models
  • Generate CRUD UI from JSON definitions
  • Handle module-specific logic with conditional checks

4. Three-Step Deployment Process

1. Database Setup: Run external database script with root privileges
2. Configuration Encryption: Encrypt sensitive data and update configs
3. Application Build: Run main build script without elevated privileges

Implementation Template

setup_database.sh

#!/usr/bin/env bash
set -e
DB_NAME="your_db"
DB_USER="your_user" 
DB_PASS="secure_password"

mysql -u root -e "CREATE DATABASE IF NOT EXISTS ${DB_NAME} CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;"
mysql -u root -e "CREATE USER IF NOT EXISTS '${DB_USER}'@'localhost' IDENTIFIED BY '${DB_PASS}';"
mysql -u root -e "GRANT ALL PRIVILEGES ON ${DB_NAME}.* TO '${DB_USER}'@'localhost';"
mysql -u root -e "FLUSH PRIVILEGES;"

encrypt_password.py

#!/usr/bin/env python3
import json
import sys

def encrypt_password(password, key):
    # Use application-specific encryption library
    # Provide fallback if library unavailable
    pass

# Read config, encrypt password, update config file

build.sh (modular version)

#!/usr/bin/env bash
set -e

MODULES=("appbase" "rbac" "contract_management")

for modulename in "${MODULES[@]}"; do
    echo "Processing ${modulename}..."
    # Uniform processing logic for all modules
done

Benefits

• Security: No root privileges needed during application build
• Maintainability: Single code path for all modules
• Reliability: Clear separation of concerns reduces failure points
• Reusability: Pattern applies to any multi-module application

Common Pitfalls

• Forgetting to update config files after password encryption
• Not handling module-specific installation differences (git vs local copy)
• Assuming MySQL connection details are always available during build
• Missing error handling for missing model or JSON files
• Case sensitivity issues: On Linux systems, Python import statements are case-sensitive. from appPublic.jsonconfig will fail if the actual file is jsonConfig.py. Always verify exact module names by inspecting the source repository structure.

Sage Module Build Pattern

For Sage platform modules, follow this standard build.sh structure:

Standard Sage build.sh Template

#!/bin/bash
set -e

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

# Find Sage root (try multiple candidates)
SAGE_ROOT=""
for candidate in "$SCRIPT_DIR/../.." "$HOME/repos/sage" "$HOME/sage"; do
    if [ -d "$candidate/wwwroot" ] && [ -d "$candidate/py3/bin" ]; then
        SAGE_ROOT="$(cd "$candidate" && pwd)"
        break
    fi
done

if [ -z "$SAGE_ROOT" ]; then
    echo "ERROR: Sage root not found"
    exit 1
fi

echo "Sage root: $SAGE_ROOT"

# Generate DDL from model JSON files
if [ -d "$SCRIPT_DIR/models" ]; then
    echo "Generating DDL from models..."
    cd "$SCRIPT_DIR/models"
    if ls *.json 1>/dev/null 2>&1; then
        "$SAGE_ROOT/py3/bin/json2ddl" mysql . > mysql.ddl.sql
        echo "DDL generated: models/mysql.ddl.sql"
    fi
fi

# Link wwwroot to Sage via symlink (not copy)
echo "Linking wwwroot to Sage..."
rm -f "$SAGE_ROOT/wwwroot/<module_name>"
ln -sf "$SCRIPT_DIR/wwwroot" "$SAGE_ROOT/wwwroot/<module_name>"

echo "<Module Name> build complete."

Key Sage Build Patterns

• Sage Root Detection: Check for both wwwroot/ and py3/bin/ directories to confirm valid Sage installation
• DDL Generation: Use $SAGE_ROOT/py3/bin/json2ddl mysql . in the models/ directory to generate SQL from JSON model definitions
• wwwroot Linking: Always use symlinks (ln -sf), never copy files - this keeps modules in sync during development
• Model Directory: JSON model files live in models/*.json, DDL output goes to models/mysql.ddl.sql
• Module Naming: Replace <module_name> with your actual module name (e.g., sage_datamart, dashboard_for_sage)

Examples from Sage Codebase

• supplychain/build.sh - includes CRUD UI generation from json/ definitions
• dashboard_for_sage/build.sh - includes pip install and API file linking