yumoqing/pbls
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
pbls/build.sh
agent.develop bc4aaa2463 fix(pbls): [deploy-Q4] build.sh gate 6.4 strip comments/COMMENT strings before scanning + table count 36 to 37 
Defect (deploy_test round 3, Q4 / deploy-evidence E15):
1. Gate 6.4 grepped the raw DDL for dialect keywords and was killed by 3 false positives
   that are pure documentation text: L6 header line comment, L220 business word
   "references" inside a COMMENT string, L891 contract self-check block that literally
   lists FOREIGN KEY / REFERENCES / ENUM( / SERIAL / BIGSERIAL / nextval / TIMESTAMP.
   The gate lost all discriminating power (false positive die) and real violations
   would be hidden in the noise.
2. Table count assertion 36 != actual DDL 37 (grep -c CREATE TABLE IF NOT EXISTS = 37;
   the 37th table is pbl_compile_task, pbl_compiler domain 4 to 5). Related: the
   AUTOINC assertion used a literal multi-space pattern and missed the UNSIGNED single
   space variant of pbl_compile_task (counted 36, accidentally matching the old assert);
   L210 post-create information_schema assertion was also 36.

Fix:
- Add scripts/ddl_gate_check.py: strip_sql_noise() single-pass char scanner removing
  block comments, -- / # line comments (-- must be followed by whitespace, so a--b is
  safe) and single/double quoted string literals (with doubled and backslash escapes)
  replaced by a placeholder, so COMMENT strings and DEFAULT strings no longer interfere.
  Assertions run only on the denoised DDL body: zero dialect keyword hits (prints
  keyword + line number + line text), CREATE TABLE count = 37 (plus duplicate table and
  non pbl_ prefix checks), BIGINT [UNSIGNED] NOT NULL AUTO_INCREMENT = 37 (table option
  AUTO_INCREMENT=n excluded first). Ships --selftest with 9 minimal unit checks (no
  pytest needed) and --dump-body for debugging; expected values are module constants.
- Rewrite build.sh gate 6.4: step 4a runs --selftest first (if the scanner is not
  trustworthy its verdict is not accepted), step 4b runs the gate with
  --expect-tables 37 --expect-autoinc 37; return codes captured explicitly (not via
  "if ! cmd" which resets $? to 0); missing scanner file dies and forbids falling back
  to raw grep. Header comment and L210 actual table assertion synced 36 to 37.
- pbls_tables.sql trailing contract self-check block: total tables 36 to 37
  (pbl_compiler 4 to 5, new pbl_compile_task); the zero-hit section now describes the
  forbidden constructs in prose instead of listing SQL keywords verbatim (defence in
  depth against a future regression to naive grep).
- projects/pbls/env/test.json gate.build_sh text "36 tables" to "37 tables"
  (json.load validity self-check passed).

Verification (real local run output, details in
projects/pbls/docs/02-develop/dev-notes-deploy-q4-build-gate64.md):
- --selftest 9/9 PASS, including a negative fixture proving real violations are still
  caught, idempotency of denoising, and that -- / # inside strings do not truncate.
- Real DDL: denoising strips 10974 chars / 89 line comments / 493 string literals;
  dialect zero hits, CREATE TABLE = 37, AUTO_INCREMENT = 37, RESULT PASS rc=0.
- build.sh gate sections 0-4 sliced into a temporary harness and executed as a whole
  inside apps/pbls, rc=0 (6.1/6.2/6.4 all pass). Sections 7+ not entered: no MariaDB,
  no venv deps, port 9300 unavailable - honestly declared in the dev notes.
- bash -n build.sh passes; no live assertion still says 36 (only 2 comment lines).

No real table structure, app entry, conf/config.json, requirements or modules/pbl_*
were modified.
2026-09-25 08:37:51 +08:00

261 lines
15 KiB
Bash
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

#!/usr/bin/env bash
# =====================================================================
# PBL Agent OS 验证原型（Phase 0/1）—— 一键部署脚本（test 环境）
# 端口唯一事实源：projects/pbls/env/test.json（9300）；本脚本不硬编码端口，从 env 读取
# 门禁：建表 37 张 + 9 项种子全绿 + self_check PASS 74/74 + py_compile 全通过
# =====================================================================
set -euo pipefail
APP_NAME="pbls"
APP_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
WS_DIR="$(cd "${APP_DIR}/../.." && pwd)"
ENV_FILE="${WS_DIR}/projects/pbls/env/test.json"
DDL_FILE="${APP_DIR}/scripts/ddl/pbls_tables.sql"
VENV_DIR="${APP_DIR}/venv"
PY="${VENV_DIR}/bin/python"
log() { echo "[build.sh][$(date '+%F %T')] $*"; }
die() { echo "[build.sh][FATAL] $*" >&2; exit 1; }
# ---- 0. 前置检查：env 唯一事实源必须存在且无占位 ----
[ -f "${ENV_FILE}" ] || die "缺少环境事实源 ${ENV_FILE}"
if grep -Eq '待明确|TBD|占位|CHANGE_ME|xxx' "${ENV_FILE}"; then
die "${ENV_FILE} 含占位/待明确字段，禁止部署"
fi
read_env() { python3 -c "import json,sys;print(json.load(open('${ENV_FILE}'))$1)"; }
PORT="$(read_env "['port']")"
DB_HOST="$(read_env "['db']['host']")"
DB_PORT="$(read_env "['db']['port']")"
DB_USER="$(read_env "['db']['user']")"
DB_PASS="$(read_env "['db']['password']")"
DB_NAME="$(read_env "['db']['dbname']")"
DEPLOY_PATH="$(read_env "['deploy']['path']")"
log "env 事实源: port=${PORT} db=${DB_NAME}@${DB_HOST}:${DB_PORT} deploy=${DEPLOY_PATH}"
# ---- 1. 端口一致性核验（conf/config.json 必须与 env 同值）----
CONF_PORT="$(python3 -c "import json;print(json.load(open('${APP_DIR}/conf/config.json'))['port'])")"
[ "${CONF_PORT}" = "${PORT}" ] || die "端口不一致: conf=${CONF_PORT} env=${PORT}"
log "端口一致性核验通过: ${PORT}"
# ---- 2. 应用仓库不得另存 env/（端口唯一事实源在项目侧）----
[ ! -d "${APP_DIR}/env" ] || die "应用仓库存在 env/ 目录，违反端口唯一事实源规则"
log "应用仓库无 env/ 目录，核验通过"
# ---- 3. 入口硬门禁核验（6.1 / 6.2）----
ENTRY="${APP_DIR}/app/${APP_NAME}.py"
[ -f "${ENTRY}" ] || die "缺少入口 ${ENTRY}"
grep -q 'from ahserver.webapp import webapp' "${ENTRY}" || die "入口未使用 ahserver.webapp.webapp"
grep -q 'webapp(init)' "${ENTRY}" || die "入口未调用 webapp(init)"
if grep -Eq 'Flask|FastAPI|Django|http\.server|jsonify|@app\.route' "${ENTRY}"; then
die "入口含禁用框架关键字（Flask/FastAPI/Django/http.server/jsonify/@app.route）"
fi
log "硬门禁 6.1 通过: ahserver.webapp + webapp(init)，无禁用框架"
# ServerEnv() 行号必须早于 load_rbac() 行号
ENV_LINE="$(grep -n 'ServerEnv()' "${ENTRY}" | head -1 | cut -d: -f1)"
RBAC_LINE="$(grep -n 'load_rbac()' "${ENTRY}" | head -1 | cut -d: -f1)"
[ -n "${ENV_LINE}" ] && [ -n "${RBAC_LINE}" ] || die "入口缺 ServerEnv() 或 load_rbac() 显式调用"
[ "${ENV_LINE}" -lt "${RBAC_LINE}" ] || die "ServerEnv() 必须早于 load_rbac()（env 行 ${ENV_LINE} >= rbac 行 ${RBAC_LINE}）"
grep -q 'load_appbase()' "${ENTRY}" || die "入口缺 load_appbase() 显式调用"
grep -q 'load_pybricks()' "${ENTRY}" || die "入口缺 load_pybricks() 显式调用"
log "硬门禁 6.2 通过: ServerEnv()(L${ENV_LINE}) < load_rbac()(L${RBAC_LINE})，load_appbase/load_rbac/load_pybricks 均显式调用"
# ---- 4. DDL 方言核验（6.4：mariadb，BIGINT AUTO_INCREMENT，零 SERIAL/FK/ENUM/TIMESTAMP）----
# 【Q4 修复 / deploy_test 第 3 轮实测】旧版直接对 DDL **原文** grep，存在两类假阳性：
# (1) DDL 文末「契约自检」注释块为说明"零命中项"，逐字写出了
# FOREIGN KEY / REFERENCES / ENUM( / SERIAL / BIGSERIAL / nextval / TIMESTAMP
# （实测原文命中 3 行：L6 头部说明、L220 COMMENT 串内的 references、L891 契约自检块），
# 旧 `grep -Eiq` 一律触发 die —— 注释/COMMENT 串被当成真实 DDL，门禁必然误杀；
# (2) `CREATE TABLE` / `AUTO_INCREMENT` 计数同理会把注释行与 COMMENT '...' 串内字样算进去，
# 表数不可信（且旧断言 36 与 DDL 实际 37 张不符，新增 pbl_compile_task 后未同步）。
# 修复：改为「先剥 SQL 注释 + 字符串字面量（含 COMMENT 串），再在纯 DDL 正文上扫描」，
# 由 scripts/ddl_gate_check.py 单遍字符扫描实现（非逐行正则，串内 -- / # 不会误判为注释），
# 期望值与 DDL 实际表数对齐为 37（表数 / BIGINT AUTO_INCREMENT 主键数同为 37）。
[ -f "${DDL_FILE}" ] || die "缺少 DDL ${DDL_FILE}"
DDL_GATE="${APP_DIR}/scripts/ddl_gate_check.py"
[ -f "${DDL_GATE}" ] || die "缺少 DDL 门禁扫描器 ${DDL_GATE}（6.4 依赖，禁止回退到原文 grep）"
# 4a. 扫描器自身可信度：先跑内置最小单测（剥注释/剥串/幂等/真实违规不漏报）
SELFTEST_RC=0
python3 "${DDL_GATE}" --selftest || SELFTEST_RC=$?
[ "${SELFTEST_RC}" = "0" ] || die "DDL 门禁扫描器自检未通过（--selftest rc=${SELFTEST_RC}），扫描结果不可信，终止部署"
# 4b. 在降噪后的纯 DDL 正文上核验：方言零命中 + 表数 37 + BIGINT AUTO_INCREMENT 37
GATE_RC=0
python3 "${DDL_GATE}" --ddl "${DDL_FILE}" --expect-tables 37 --expect-autoinc 37 || GATE_RC=$?
[ "${GATE_RC}" = "0" ] || die "硬门禁 6.4 未通过（ddl_gate_check.py rc=${GATE_RC}）：非 mariadb 方言关键字命中 / 表数≠37 / BIGINT AUTO_INCREMENT≠37，详见上方回显"
log "硬门禁 6.4 通过: mariadb 方言（剥注释+COMMENT 串后零命中），37 表 / 37 处 BIGINT AUTO_INCREMENT"
# ---- 5. 依赖声明核验（6.5）----
for dep in ahserver sqlor apppublic appbase rbac PyMySQL; do
grep -qi "^${dep}" "${APP_DIR}/requirements.txt" || die "requirements.txt 缺依赖 ${dep}"
done
log "硬门禁 6.5 通过: ahserver/sqlor/apppublic/appbase/rbac/PyMySQL 均已声明"
# ---- 6. website.processors / indexes 核验（缺一会导致 / 返回 500）----
python3 - <<'PYEOF'
import json, sys
conf = json.load(open('conf/config.json'))
procs = dict((p[0], p[1]) for p in conf['website']['processors'])
for ext, name in (('.tmpl', 'tmpl'), ('.ui', 'bui'), ('.dspy', 'dspy')):
assert procs.get(ext) == name, 'processors 缺 %s->%s，实际 %r' % (ext, name, procs.get(ext))
assert 'index.ui' in conf['website']['indexes'], 'indexes 缺 index.ui'
print('[build.sh] website 核验通过: processors=%s indexes=%s' % (procs, conf['website']['indexes']))
PYEOF
# ---- 7. venv + 依赖安装 ----
if [ ! -x "${PY}" ]; then
log "创建 venv: ${VENV_DIR}"
python3 -m venv "${VENV_DIR}"
fi
"${PY}" -m pip install --upgrade pip wheel setuptools
"${PY}" -m pip install -r "${APP_DIR}/requirements.txt"
# ---- 7b. 测试前置依赖（DISC.2）----
# 运行时依赖（requirements.txt）刻意不含 pytest；测试依赖单列 requirements-test.txt，
# 由本步装进同一个 venv，使
# source venv/bin/activate && python3 -m pytest tests/test_m5a_idempotency.py -v
# 在**不手工注入 PYTHONPATH** 的前提下可直接执行（此前靠 PYTHONPATH=<ws>/.pylibs
# 临时注入跑通，换机器/换角色即失效，test 角色会误判为环境不可用）。
# 只要运行时依赖的场景（如生产镜像）用 PBL_SKIP_TEST_DEPS=1 显式跳过。
REQ_TEST="${APP_DIR}/requirements-test.txt"
if [ "${PBL_SKIP_TEST_DEPS:-0}" = "1" ]; then
log "PBL_SKIP_TEST_DEPS=1：跳过测试前置依赖（venv 内将没有 pytest，测试角色需自行引导）"
elif [ ! -f "${REQ_TEST}" ]; then
die "缺测试依赖清单 ${REQ_TEST}（DISC.2 要求显式声明；确要跳过请设 PBL_SKIP_TEST_DEPS=1）"
else
log "安装测试前置依赖: ${REQ_TEST}"
"${PY}" -m pip install -r "${REQ_TEST}"
# 自证：装完必须真能 import + 跑 --version，否则本步等于没做（不让注释与事实矛盾）
"${PY}" -c 'import pytest' >/dev/null 2>&1 \
|| die "安装后 venv 内仍无法 import pytest（${PY}）"
log "7b 自证通过: $("${PY}" -m pytest --version 2>&1 | head -1) @ ${PY}"
fi
# ---- 8. 本地安装 15 个 pbl_* 模块 + 复用模块（可编辑模式）----
for mod_dir in "${WS_DIR}"/modules/pbl_*; do
[ -d "${mod_dir}" ] || continue
if [ -f "${mod_dir}/pyproject.toml" ] || [ -f "${mod_dir}/setup.py" ]; then
log "安装模块 $(basename "${mod_dir}")"
"${PY}" -m pip install -e "${mod_dir}"
else
export PYTHONPATH="${mod_dir}:${PYTHONPATH:-}"
log "模块 $(basename "${mod_dir}") 无打包描述，加入 PYTHONPATH"
fi
done
# ---- 8b. M9 覆盖层页面壳生成（构建产物，不入库）+ 一致性门禁 ----
# 页面壳由 pbl_scense_ext 的生成器产出；产物目录 wwwroot/pbl_overlay/ 在模块仓
# .gitignore 中被排除，因此必须在构建期生成，否则新克隆/新部署环境该目录为空、
# /pbl_overlay/<key>.ui 导航全量 404。位置：模块安装之后、模块 wwwroot 软链（紧随其后的
# 「8c. 模块 wwwroot 挂载」段）之前——生成物落在模块仓，必须再经 8c 挂载到应用 wwwroot
# 才能在应用侧 URL /pbl_scense_ext/pbl_overlay/<key>.ui 上取到。
OVERLAY_GEN="${WS_DIR}/modules/pbl_scense_ext/scripts/gen_overlay_pages.py"
[ -f "${OVERLAY_GEN}" ] || die "缺少覆盖层页面壳生成器: ${OVERLAY_GEN}（modules/pbl_scense_ext 未就位）"
log "生成 PBL 覆盖层页面壳: ${OVERLAY_GEN}"
# 注意：不能用 `if ! cmd; then die "... rc=$?"` —— 进入分支时 $? 已被 if 取反为 0，
# 打印出的 rc 恒为 0 会误导排障；改为显式捕获真实返回码。
GEN_RC=0
"${PY}" "${OVERLAY_GEN}" || GEN_RC=$?
if [ "${GEN_RC}" -ne 0 ]; then
die "覆盖层页面壳生成失败（gen_overlay_pages.py rc=${GEN_RC}）：wwwroot/pbl_overlay/ 将为空，/pbl_overlay/<key>.ui 全量 404，终止部署"
fi
log "覆盖层页面壳门禁校验（gen_overlay_pages.py --check）"
CHK_RC=0
"${PY}" "${OVERLAY_GEN}" --check || CHK_RC=$?
if [ "${CHK_RC}" -ne 0 ]; then
die "覆盖层页面壳门禁校验未通过（--check rc=${CHK_RC}）：i18n zh/en key 不一致，或页面壳与 PAGES 注册表不一致，终止部署"
fi
log "覆盖层页面壳生成 + 门禁校验通过（12 个页面壳 / i18n key 一致）"
# ---- 8c. 模块 wwwroot 挂载（软链）+ 应用侧可达性门禁（QC 退回意见 #1/#2）----
# 上一版只做了 8b「生成」，生成物落在模块仓 modules/<m>/wwwroot/ 下，但本脚本从未把
# 模块 wwwroot 挂进应用 wwwroot —— 应用 website.root=wwwroot 只指向应用自身目录，
# 因此 build.sh 跑完后应用侧依然取不到 /pbl_scense_ext/pbl_overlay/<key>.ui（可达链路
# 断在挂载环节，全量 404 并未消除）。此处按 web-application-spec「挂载模块 9 步流程」
# 第 5 步（wwwroot 软链，非 cp，保持与模块仓单一副本同步）补齐，并立即以应用侧路径
# 实测 12 个页面壳数量作为门禁，rc≠0 终止部署。
LINKED_MODULES=""
for mod_dir in "${WS_DIR}"/modules/pbl_*; do
[ -d "${mod_dir}/wwwroot" ] || continue
mod_name="$(basename "${mod_dir}")"
www_link="${APP_DIR}/wwwroot/${mod_name}"
# 已存在但不是软链（应用仓自带同名真实目录）时不覆盖，直接终止让人工处置，
# 避免 ln -sfn 把软链建进既有目录里、或静默删掉应用仓跟踪文件。
if [ -e "${www_link}" ] && [ ! -L "${www_link}" ]; then
die "wwwroot 挂载冲突：${www_link} 已存在且不是软链（应用仓自带目录？），请人工处置后重试"
fi
ln -sfn "${mod_dir}/wwwroot" "${www_link}"
LINKED_MODULES="${LINKED_MODULES} ${mod_name}"
done
[ -n "${LINKED_MODULES}" ] || die "未发现任何含 wwwroot/ 的 pbl_* 模块，模块 wwwroot 挂载为空"
log "模块 wwwroot 软链挂载完成:${LINKED_MODULES}"
# 挂载后应用侧可达性门禁：覆盖层页面壳必须能通过应用 wwwroot 路径取到
APP_OVERLAY_DIR="${APP_DIR}/wwwroot/pbl_scense_ext/pbl_overlay"
APP_UI_COUNT=0
if [ -d "${APP_OVERLAY_DIR}" ]; then
APP_UI_COUNT="$(find -L "${APP_OVERLAY_DIR}" -maxdepth 1 -name '*.ui' | wc -l | tr -d '[:space:]')"
fi
if [ "${APP_UI_COUNT}" != "12" ]; then
die "应用侧覆盖层页面壳应为 12 个，实际 ${APP_UI_COUNT}（${APP_OVERLAY_DIR}）：模块 wwwroot 未挂载成功或生成器未产出，/pbl_scense_ext/pbl_overlay/<key>.ui 将全量 404，终止部署"
fi
log "应用侧可达性核验通过: ${APP_OVERLAY_DIR}/*.ui = ${APP_UI_COUNT}"
# ---- 9. 语法验证：全部 .py 跑 py_compile（QC 退回意见 #16）----
log "py_compile 全量语法验证"
FAIL_SYNTAX=0
while IFS= read -r pyfile; do
if ! "${PY}" -m py_compile "${pyfile}"; then
echo "[build.sh][SYNTAX FAIL] ${pyfile}"
FAIL_SYNTAX=1
fi
done < <(find "${APP_DIR}/app" "${WS_DIR}/modules/pbl_common" "${WS_DIR}/modules/pbl_appcodes" -name '*.py' -not -path '*/venv/*' 2>/dev/null)
[ "${FAIL_SYNTAX}" = "0" ] || die "存在语法错误文件，终止部署"
log "py_compile 全部通过"
# ---- 10. 建表（mariadb，显式 TCP，禁 socket）----
log "执行 DDL: ${DDL_FILE}"
mysql --protocol=TCP -h"${DB_HOST}" -P"${DB_PORT}" -u"${DB_USER}" -p"${DB_PASS}" \
--default-character-set=utf8mb4 "${DB_NAME}" < "${DDL_FILE}"
ACTUAL_TABLES="$(mysql --protocol=TCP -h"${DB_HOST}" -P"${DB_PORT}" -u"${DB_USER}" -p"${DB_PASS}" \
-N -B -e "SELECT COUNT(*) FROM information_schema.tables WHERE table_schema='${DB_NAME}' AND table_name LIKE 'pbl\\_%'" "${DB_NAME}")"
# Q4 修复：与 DDL 实际表数对齐（36 → 37，新增 pbl_compile_task）
[ "${ACTUAL_TABLES}" = "37" ] || die "建表数应为 37，实际 ${ACTUAL_TABLES}"
log "建表完成: pbl_* 共 ${ACTUAL_TABLES} 张"
# ---- 11. 枚举数量契约自检（QC 退回意见 #15：必须真实可运行输出）----
"${PY}" -m pbl_appcodes.self_check | tee "${APP_DIR}/logs/self_check.log"
grep -q 'SELF_CHECK pbl_appcodes: PASS 74/74' "${APP_DIR}/logs/self_check.log" \
|| die "pbl_appcodes 数量契约自检未通过"
# ---- 12. 9 项种子数据幂等注入（重复执行零差异）----
log "种子注入（第 1 轮）"
"${PY}" - <<'PYEOF'
import json
from pbl_appcodes.seed import seed_all
s1 = seed_all()
print('[seed round1]', json.dumps(s1, ensure_ascii=False, sort_keys=True))
s2 = seed_all()
print('[seed round2]', json.dumps(s2, ensure_ascii=False, sort_keys=True))
assert s2['groups_inserted'] == 0 and s2['groups_updated'] == 0, '重复执行组有差异'
assert s2['items_inserted'] == 0 and s2['items_updated'] == 0, '重复执行项有差异'
assert s2['idempotent'] is True, '幂等校验失败'
print('[seed] 幂等零差异校验通过: 8 组 74 项')
PYEOF
# ---- 13. 启动应用（唯一入口 + 唯一端口）----
log "启动 ${APP_NAME} 于端口 ${PORT}"
mkdir -p "${APP_DIR}/logs" "${APP_DIR}/run"
nohup "${PY}" "${ENTRY}" > "${APP_DIR}/logs/${APP_NAME}.out" 2>&1 &
echo $! > "${APP_DIR}/run/${APP_NAME}.pid"
sleep 5
# ---- 14. 健康检查 ----
HTTP_CODE="$(curl -s -o /dev/null -w '%{http_code}' "http://127.0.0.1:${PORT}/api/public/health" || echo 000)"
log "健康检查 HTTP ${HTTP_CODE}"
[ "${HTTP_CODE}" = "200" ] || die "健康检查失败（HTTP ${HTTP_CODE}），查看 ${APP_DIR}/logs/${APP_NAME}.out"
log "BUILD SUCCESS: ${APP_NAME} 已运行于 http://127.0.0.1:${PORT}/ （部署路径 ${DEPLOY_PATH}）"
Reference in New Issue View Git Blame Copy Permalink