chore: add production health check automation

This commit is contained in:
Codex
2026-05-01 08:35:06 +08:00
parent f616a0017e
commit beb14828ca
5 changed files with 580 additions and 3 deletions
+140
View File
@@ -0,0 +1,140 @@
# AI合同平台 运维手册(2026-05-01)
> 适用范围:试点生产环境 `192.168.3.90`。本文档记录当前可执行的生产巡检、密码轮换、备份恢复、依赖治理和故障处理口径。
---
## 一、生产入口
| 项目 | 地址 |
|---|---|
| 前端 | `http://192.168.3.90:3003` |
| 后端 | `http://192.168.3.90:8001` |
| 健康检查 | `http://192.168.3.90:8001/health` |
| Readiness | `http://192.168.3.90:8001/api/v1/system/readiness` |
| 试点准备页 | `http://192.168.3.90:3003/pilot-readiness` |
| 服务器项目目录 | `/opt/ai-contract-platform` |
---
## 二、生产健康巡检
本机执行:
```powershell
powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-production-health-check.ps1
```
默认产物目录:
```text
project-docs/qa-artifacts/phase5-production-health/<timestamp>/
```
巡检内容:
1. 通过 SSH 读取远端 Docker 容器状态。
2. 检查后端 `8001` 和前端 `3003` 端口。
3. 检查 `/health`。
4. 检查 `/api/v1/system/readiness`。
5. 检查 `/login` 和 `/pilot-readiness`。
6. 验证默认管理员初始密码已被拒绝。
PASS 标准:
1. required 检查全部通过。
2. readiness 为 `ready`。
3. readiness `warnings=0`。
4. readiness `password_rotated=true`。
5. 默认管理员初始密码登录返回 401。
---
## 三、管理员密码口径
1. 管理员真实密码不得写入 Git、文档、脚本、`.env` 或日志。
2. 自动化 E2E 需要管理员登录时,只通过当前终端环境变量提供:
```powershell
$env:AI_CONTRACT_ADMIN_PASSWORD = '<由管理员输入的实际密码>'
```
3. 默认初始密码只能用于验证“已经失效”,不能用于生产登录。
4. 密码轮换脚本:
```powershell
powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\rotate-admin-password.ps1 -ApiBase 'http://192.168.3.90:8001/api/v1'
```
---
## 四、备份恢复口径
阶段 5 下一步将新增 `scripts/run-backup-drill.ps1`。当前原则先固定:
1. 生产备份可以读取生产数据。
2. 恢复验证必须在隔离目录或临时数据库执行。
3. 禁止把恢复演练直接覆盖生产库。
4. 每次备份必须记录备份路径、文件大小、生成时间和校验方式。
5. 每次恢复演练必须有独立日志和 PASS/FAIL 结论。
---
## 五、依赖治理口径
前端:
```powershell
Push-Location frontend
npm audit --json
npm run lint
npm run build
Pop-Location
```
后端:
```powershell
backend\.venv\Scripts\python.exe -m pip check
backend\.venv\Scripts\python.exe -m pytest -q backend/tests
```
门禁:
1. high/critical 依赖漏洞必须为 0。
2. moderate 漏洞必须修复或写明延期理由、影响范围、替代控制措施。
3. 依赖升级后必须复跑构建、测试和关键 E2E。
---
## 六、常见故障处理
查看容器状态:
```bash
cd /opt/ai-contract-platform
docker compose -f docker-compose.server.yml ps
```
查看后端日志:
```bash
cd /opt/ai-contract-platform
docker compose -f docker-compose.server.yml logs --tail=200 backend
```
查看前端日志:
```bash
cd /opt/ai-contract-platform
docker compose -f docker-compose.server.yml logs --tail=200 frontend
```
重启服务:
```bash
cd /opt/ai-contract-platform
docker compose -f docker-compose.server.yml up -d backend frontend
```
重启后必须重新执行生产健康巡检。
@@ -14,11 +14,11 @@
| 阶段 2:深度开发与交付体验补强 | PASS | `project-docs/阶段2-深度开发与交付体验补强执行记录-2026-05-01.md` |
| 阶段 3:试点交付准备 | PASS | `project-docs/阶段3-试点交付准备执行记录-2026-05-01.md` |
| 阶段 4:试点运行监控与安全加固 | PASS | `project-docs/阶段4-试点运行监控与安全加固执行记录-2026-05-01.md` |
| 阶段 5:生产运行、安全、监控、备份与依赖治理 | READY | `project-docs/阶段5-生产运行安全监控备份与依赖治理方案-2026-05-01.md` |
| 阶段 5:生产运行、安全、监控、备份与依赖治理 | IN PROGRESS | `project-docs/阶段5-生产运行安全监控备份与依赖治理执行记录-2026-05-01.md` |
阶段 1 已完成源码闭环与生产入口闭环,生产无人 E2E `18/18` 通过。阶段 2 已完成 Review 重试、Knowledge Base 文档重试、Review -> Chat -> Contract 上下文联动、首页交付工作台和阶段 2 自动化无人闭环,本地与生产 E2E 均 `7/7` 通过。阶段 3 已完成系统 readiness、试点交付准备页、默认账号说明、演示样本数据、依赖健康门禁和阶段 3 自动化无人闭环,本地与生产 E2E 均 `5/5` 通过。
当前后续重点为“生产运行、安全、监控、备份与依赖治理”:默认密码轮换已完成,下一阶段优先建立生产健康巡检、备份恢复演练、依赖治理、安全基线和生产回归闭环。
当前后续重点为“生产运行、安全、监控、备份与依赖治理”:默认密码轮换已完成,阶段 5 Step 1 生产健康巡检自动化已完成并 PASS,下一步进入备份恢复演练。
---
@@ -210,4 +210,4 @@ AI合同平台当前已经不是从零建设阶段。登录、合同审查、Rev
## 六、当前一句话结论
AI合同平台阶段 1、阶段 2 和阶段 3 均已完成闭环交付验证,阶段 4 已完成默认管理员密码轮换闭环,当前准备进入阶段 5:生产运行、安全、监控、备份与依赖治理。
AI合同平台阶段 1、阶段 2 和阶段 3 均已完成闭环交付验证,阶段 4 已完成默认管理员密码轮换闭环,阶段 5 已启动并完成生产健康巡检自动化,下一步进入备份恢复演练。
@@ -0,0 +1,97 @@
# 阶段 5:生产运行、安全、监控、备份与依赖治理执行记录(2026-05-01)
> 当前目标:阶段 5 不再继续补主业务流程,而是建立生产运行、安全、监控、备份和依赖治理的可复跑闭环。
---
## 一、当前执行结论
| 主线 | 状态 | 备注 |
|---|---|---|
| gstack 安装校验 | PASS | 自动化执行前已返回 `GSTACK_OK` |
| 代码提交与推送 | PASS | 已推送到 `origin/main`,远端为 `192.168.3.90` 仓库 |
| Step 1:生产健康巡检自动化 | PASS | `scripts/run-production-health-check.ps1` 已实现并完成生产巡检 |
| Step 2:备份与恢复演练 | PENDING | 下一步进入 |
| Step 3:依赖治理专项 | PENDING | 待备份基线后处理 |
| Step 4:安全基线检查 | PENDING | 将复用 Step 1 readiness 与默认密码验证 |
| Step 5:生产回归闭环 | PENDING | 待运维加固动作后复跑阶段 1/2/3 |
阶段 5 Step 1 当前判定:PASS。
---
## 二、Step 1 生产健康巡检自动化
新增脚本:
```powershell
scripts\run-production-health-check.ps1
```
默认检查目标:
| 项目 | 默认值 |
|---|---|
| 服务器 | `192.168.3.90` |
| 后端入口 | `http://192.168.3.90:8001` |
| 前端入口 | `http://192.168.3.90:3003` |
| 远端项目目录 | `/opt/ai-contract-platform` |
| Compose 文件 | `docker-compose.server.yml` |
本次巡检产物:
```text
project-docs/qa-artifacts/phase5-production-health/2026-05-01T08-32-50/
```
说明:`qa-artifacts` 属于本地/服务器运行证据,已从 Git 版本化中排除,避免把截图、日志和压缩包推入源码仓库。
---
## 三、本次生产巡检结果
| 检查项 | 结果 | 证据 |
|---|---|---|
| Docker 容器状态 | PASS | backend/frontend 可读取且运行中 |
| 后端端口 `8001` | PASS | TCP 可连接 |
| 前端端口 `3003` | PASS | TCP 可连接 |
| 后端 `/health` | PASS | HTTP 200 |
| `/api/v1/system/readiness` | PASS | `ready`、`warnings=0`、`password_rotated=true` |
| 前端 `/login` | PASS | HTTP 200 |
| 前端 `/pilot-readiness` | PASS | HTTP 200 |
| 默认管理员初始密码拒绝 | PASS | 登录返回 401 |
汇总文件:
```text
production-health-summary.json
production-health-summary.md
container-status.txt
health.json
readiness.json
default-password-login.json
```
---
## 四、脚本设计口径
1. 不读取、不写入管理员真实密码。
2. 默认只验证初始密码已失效。
3. SSH 只读取容器状态,不执行重启、删除、迁移等破坏性动作。
4. 所有检查结果写入 JSON 和 Markdown,便于后续巡检、监控、告警和审计复用。
5. 任何 required 检查失败时脚本退出失败。
---
## 五、下一步
阶段 5 下一步进入 Step 2:备份与恢复演练。
优先交付:
1. `scripts/run-backup-drill.ps1`
2. PostgreSQL 备份文件生成和校验
3. 部署配置、脚本、文档备份
4. 隔离恢复可用性验证
5. 备份恢复演练记录
@@ -172,6 +172,13 @@ $env:AI_CONTRACT_ADMIN_PASSWORD = '<管理员实际密码>'
2. `project-docs/阶段5-生产运行安全监控备份与依赖治理执行记录-2026-05-01.md`
3. `project-docs/运维手册-2026-05-01.md`
当前执行状态:
- Step 1 已完成:`scripts/run-production-health-check.ps1`
- 生产巡检结果:PASS
- 巡检产物:`project-docs/qa-artifacts/phase5-production-health/2026-05-01T08-32-50/`
- 下一步:Step 2 备份与恢复演练
---
## 七、一句话结论
+333
View File
@@ -0,0 +1,333 @@
param(
[string]$ServerHost = "192.168.3.90",
[string]$BackendBaseUrl = "http://192.168.3.90:8001",
[string]$FrontendBaseUrl = "http://192.168.3.90:3003",
[string]$SshUser = "root",
[string]$SshKeyPath = "$HOME\.ssh\water_audit_ed25519",
[string]$RemoteProjectPath = "/opt/ai-contract-platform",
[string]$ComposeFile = "docker-compose.server.yml",
[string]$ArtifactRoot = "",
[string]$AdminAccount = "admin@example.com",
[string]$DefaultPassword = "123456",
[switch]$SkipSsh
)
$ErrorActionPreference = "Stop"
$repoRoot = Resolve-Path (Join-Path $PSScriptRoot "..")
$timestamp = Get-Date -Format "yyyy-MM-ddTHH-mm-ss"
if ([string]::IsNullOrWhiteSpace($ArtifactRoot)) {
$ArtifactRoot = Join-Path $repoRoot "project-docs\qa-artifacts\phase5-production-health"
}
$artifactDir = Join-Path $ArtifactRoot $timestamp
$summaryJson = Join-Path $artifactDir "production-health-summary.json"
$summaryMd = Join-Path $artifactDir "production-health-summary.md"
$containerStatusPath = Join-Path $artifactDir "container-status.txt"
$readinessPath = Join-Path $artifactDir "readiness.json"
$healthPath = Join-Path $artifactDir "health.json"
$defaultLoginPath = Join-Path $artifactDir "default-password-login.json"
New-Item -ItemType Directory -Force -Path $artifactDir | Out-Null
if (-not (Test-Path "$HOME\.Codex\skills\gstack\bin")) {
throw "gstack is required for all AI-assisted work in this repo."
}
Add-Type -AssemblyName System.Net.Http
$checks = New-Object System.Collections.Generic.List[object]
function Add-Check {
param(
[string]$Key,
[string]$Label,
[ValidateSet("pass", "fail", "warn")]
[string]$Status,
[string]$Detail,
[bool]$Required = $true,
[hashtable]$Meta = @{}
)
$checks.Add([ordered]@{
key = $Key
label = $Label
status = $Status
required = $Required
detail = $Detail
meta = $Meta
}) | Out-Null
}
function ConvertTo-SafeFileName {
param([string]$Value)
return ($Value -replace '[^a-zA-Z0-9._-]', '-').Trim("-")
}
function Save-Json {
param(
[string]$Path,
[object]$Value
)
$Value | ConvertTo-Json -Depth 12 | Set-Content -Encoding UTF8 -Path $Path
}
function Invoke-HttpProbe {
param(
[string]$Method,
[string]$Url,
[object]$Body = $null
)
$handler = New-Object System.Net.Http.HttpClientHandler
$client = New-Object System.Net.Http.HttpClient($handler)
$client.Timeout = [TimeSpan]::FromSeconds(15)
try {
$httpMethod = New-Object System.Net.Http.HttpMethod($Method)
$request = New-Object System.Net.Http.HttpRequestMessage($httpMethod, $Url)
if ($null -ne $Body) {
$json = $Body | ConvertTo-Json -Depth 8 -Compress
$request.Content = New-Object System.Net.Http.StringContent($json, [System.Text.Encoding]::UTF8, "application/json")
}
$response = $client.SendAsync($request).GetAwaiter().GetResult()
$text = $response.Content.ReadAsStringAsync().GetAwaiter().GetResult()
$parsed = $null
if (-not [string]::IsNullOrWhiteSpace($text)) {
try {
$parsed = $text | ConvertFrom-Json -ErrorAction Stop
} catch {
$parsed = $null
}
}
return [ordered]@{
ok = $true
url = $Url
method = $Method
status_code = [int]$response.StatusCode
body = $parsed
body_text = if ($parsed) { $null } else { $text }
}
} catch {
return [ordered]@{
ok = $false
url = $Url
method = $Method
status_code = $null
error = $_.Exception.Message
}
} finally {
$client.Dispose()
$handler.Dispose()
}
}
function Test-TcpPort {
param(
[string]$HostName,
[int]$Port,
[int]$TimeoutMs = 3000
)
$client = New-Object System.Net.Sockets.TcpClient
try {
$async = $client.BeginConnect($HostName, $Port, $null, $null)
$connected = $async.AsyncWaitHandle.WaitOne($TimeoutMs, $false)
if (-not $connected) {
return $false
}
$client.EndConnect($async)
return $client.Connected
} catch {
return $false
} finally {
$client.Close()
}
}
function Get-UriHostAndPort {
param(
[string]$Url,
[int]$DefaultPort
)
$uri = [Uri]$Url
$port = if ($uri.IsDefaultPort) { $DefaultPort } else { $uri.Port }
return [ordered]@{
host = $uri.Host
port = $port
}
}
function Invoke-SshCommand {
param([string]$Command)
$target = "${SshUser}@${ServerHost}"
$args = @(
"-i", $SshKeyPath,
"-o", "BatchMode=yes",
"-o", "StrictHostKeyChecking=accept-new",
"-o", "ConnectTimeout=10",
$target,
$Command
)
$output = & ssh @args 2>&1
return [ordered]@{
exit_code = $LASTEXITCODE
output = ($output -join [Environment]::NewLine)
}
}
$backendTarget = Get-UriHostAndPort -Url $BackendBaseUrl -DefaultPort 80
$frontendTarget = Get-UriHostAndPort -Url $FrontendBaseUrl -DefaultPort 80
if ($SkipSsh) {
Add-Check -Key "container_status" -Label "Docker containers" -Status "warn" -Required $false -Detail "Skipped by -SkipSsh"
} elseif (-not (Test-Path $SshKeyPath)) {
Add-Check -Key "container_status" -Label "Docker containers" -Status "fail" -Detail "SSH key not found: $SshKeyPath"
} else {
$remoteCommand = "cd $RemoteProjectPath; docker compose -f $ComposeFile ps"
$sshResult = Invoke-SshCommand -Command $remoteCommand
$sshResult.output | Set-Content -Encoding UTF8 -Path $containerStatusPath
$hasBackend = $sshResult.output -match "backend"
$hasFrontend = $sshResult.output -match "frontend"
$looksRunning = $sshResult.output -match "running|Up|healthy"
if ($sshResult.exit_code -eq 0 -and $hasBackend -and $hasFrontend -and $looksRunning) {
Add-Check -Key "container_status" -Label "Docker containers" -Status "pass" -Detail "backend/frontend containers are visible and running" -Meta @{
artifact = $containerStatusPath
}
} else {
Add-Check -Key "container_status" -Label "Docker containers" -Status "fail" -Detail "Could not confirm backend/frontend running state, SSH exit=$($sshResult.exit_code)" -Meta @{
artifact = $containerStatusPath
}
}
}
foreach ($target in @(
@{ key = "backend_port"; label = "Backend port"; host = $backendTarget.host; port = $backendTarget.port },
@{ key = "frontend_port"; label = "Frontend port"; host = $frontendTarget.host; port = $frontendTarget.port }
)) {
$portOpen = Test-TcpPort -HostName $target.host -Port $target.port
if ($portOpen) {
Add-Check -Key $target.key -Label $target.label -Status "pass" -Detail "$($target.host):$($target.port) accepts TCP connections"
} else {
Add-Check -Key $target.key -Label $target.label -Status "fail" -Detail "$($target.host):$($target.port) does not accept TCP connections"
}
}
$health = Invoke-HttpProbe -Method "GET" -Url "$BackendBaseUrl/health"
Save-Json -Path $healthPath -Value $health
if ($health.ok -and $health.status_code -eq 200) {
Add-Check -Key "backend_health" -Label "Backend /health" -Status "pass" -Detail "/health returned 200" -Meta @{ artifact = $healthPath }
} else {
Add-Check -Key "backend_health" -Label "Backend /health" -Status "fail" -Detail "/health did not return 200" -Meta @{
status_code = $health.status_code
artifact = $healthPath
}
}
$readiness = Invoke-HttpProbe -Method "GET" -Url "$BackendBaseUrl/api/v1/system/readiness"
Save-Json -Path $readinessPath -Value $readiness
$readinessData = $readiness.body.data
$passwordRotated = $false
if ($readinessData -and $readinessData.pilot_assets -and $readinessData.pilot_assets.default_admin) {
$passwordRotated = [bool]$readinessData.pilot_assets.default_admin.password_rotated
}
$warnings = if ($readinessData -and $readinessData.summary) { [int]$readinessData.summary.warnings } else { -1 }
if ($readiness.ok -and $readiness.status_code -eq 200 -and $readiness.body.code -eq 0 -and $readinessData.status -eq "ready" -and $warnings -eq 0 -and $passwordRotated) {
Add-Check -Key "system_readiness" -Label "System readiness" -Status "pass" -Detail "readiness=ready, warnings=0, password_rotated=true" -Meta @{ artifact = $readinessPath }
} else {
Add-Check -Key "system_readiness" -Label "System readiness" -Status "fail" -Detail "readiness did not satisfy ready/warnings=0/password_rotated=true" -Meta @{
status_code = $readiness.status_code
readiness_status = $readinessData.status
warnings = $warnings
password_rotated = $passwordRotated
artifact = $readinessPath
}
}
foreach ($page in @("/login", "/pilot-readiness")) {
$url = "$FrontendBaseUrl$page"
$probe = Invoke-HttpProbe -Method "GET" -Url $url
$fileName = "page-$((ConvertTo-SafeFileName -Value $page)).json"
$pagePath = Join-Path $artifactDir $fileName
Save-Json -Path $pagePath -Value $probe
$key = "frontend_page_$((ConvertTo-SafeFileName -Value $page).ToLowerInvariant())"
if ($probe.ok -and $probe.status_code -eq 200) {
Add-Check -Key $key -Label "Frontend$page" -Status "pass" -Detail "$page returned 200" -Meta @{ artifact = $pagePath }
} else {
Add-Check -Key $key -Label "Frontend$page" -Status "fail" -Detail "$page did not return 200" -Meta @{
status_code = $probe.status_code
artifact = $pagePath
}
}
}
$defaultLogin = Invoke-HttpProbe -Method "POST" -Url "$BackendBaseUrl/api/v1/auth/login" -Body @{
account = $AdminAccount
password = $DefaultPassword
}
Save-Json -Path $defaultLoginPath -Value $defaultLogin
if ($defaultLogin.ok -and $defaultLogin.status_code -eq 401) {
Add-Check -Key "default_password_rejected" -Label "Default password rejection" -Status "pass" -Detail "Default admin password login returned 401" -Meta @{ artifact = $defaultLoginPath }
} else {
Add-Check -Key "default_password_rejected" -Label "Default password rejection" -Status "fail" -Detail "Default admin password was not clearly rejected" -Meta @{
status_code = $defaultLogin.status_code
artifact = $defaultLoginPath
}
}
$failedRequired = @($checks | Where-Object { $_.required -and $_.status -eq "fail" })
$warningsList = @($checks | Where-Object { $_.status -eq "warn" })
$overallStatus = if ($failedRequired.Count -gt 0) { "FAIL" } elseif ($warningsList.Count -gt 0) { "WARN" } else { "PASS" }
$summary = [ordered]@{
name = "phase5-production-health"
status = $overallStatus
generated_at = (Get-Date).ToUniversalTime().ToString("o")
artifact_dir = $artifactDir
targets = [ordered]@{
server_host = $ServerHost
backend_base_url = $BackendBaseUrl
frontend_base_url = $FrontendBaseUrl
remote_project_path = $RemoteProjectPath
compose_file = $ComposeFile
}
checks = $checks
totals = [ordered]@{
total = $checks.Count
failed_required = $failedRequired.Count
warnings = $warningsList.Count
}
}
Save-Json -Path $summaryJson -Value $summary
$markdown = New-Object System.Collections.Generic.List[string]
$markdown.Add("# Phase 5 Production Health Check") | Out-Null
$markdown.Add("") | Out-Null
$markdown.Add("- Status: $overallStatus") | Out-Null
$markdown.Add("- Generated at: $($summary.generated_at)") | Out-Null
$markdown.Add("- Backend: $BackendBaseUrl") | Out-Null
$markdown.Add("- Frontend: $FrontendBaseUrl") | Out-Null
$markdown.Add("- Artifact dir: $artifactDir") | Out-Null
$markdown.Add("") | Out-Null
$markdown.Add("| Check | Required | Status | Detail |") | Out-Null
$markdown.Add("|---|---:|---|---|") | Out-Null
foreach ($check in $checks) {
$markdown.Add("| $($check.label) | $($check.required) | $($check.status.ToUpperInvariant()) | $($check.detail) |") | Out-Null
}
$markdown.Add("") | Out-Null
$markdown.Add("Raw artifacts:") | Out-Null
$markdown.Add("") | Out-Null
$markdown.Add('- `production-health-summary.json`') | Out-Null
$markdown.Add('- `readiness.json`') | Out-Null
$markdown.Add('- `health.json`') | Out-Null
$markdown.Add('- `default-password-login.json`') | Out-Null
if (Test-Path $containerStatusPath) {
$markdown.Add('- `container-status.txt`') | Out-Null
}
$markdown | Set-Content -Encoding UTF8 -Path $summaryMd
Write-Host "PHASE5_PRODUCTION_HEALTH_$overallStatus $artifactDir"
if ($overallStatus -eq "FAIL") {
throw "Phase 5 production health check failed. See $summaryMd"
}