chore: add production health check automation
This commit is contained in:
@@ -0,0 +1,140 @@
|
||||
# AI合同平台 运维手册(2026-05-01)
|
||||
|
||||
> 适用范围:试点生产环境 `192.168.3.90`。本文档记录当前可执行的生产巡检、密码轮换、备份恢复、依赖治理和故障处理口径。
|
||||
|
||||
---
|
||||
|
||||
## 一、生产入口
|
||||
|
||||
| 项目 | 地址 |
|
||||
|---|---|
|
||||
| 前端 | `http://192.168.3.90:3003` |
|
||||
| 后端 | `http://192.168.3.90:8001` |
|
||||
| 健康检查 | `http://192.168.3.90:8001/health` |
|
||||
| Readiness | `http://192.168.3.90:8001/api/v1/system/readiness` |
|
||||
| 试点准备页 | `http://192.168.3.90:3003/pilot-readiness` |
|
||||
| 服务器项目目录 | `/opt/ai-contract-platform` |
|
||||
|
||||
---
|
||||
|
||||
## 二、生产健康巡检
|
||||
|
||||
本机执行:
|
||||
|
||||
```powershell
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-production-health-check.ps1
|
||||
```
|
||||
|
||||
默认产物目录:
|
||||
|
||||
```text
|
||||
project-docs/qa-artifacts/phase5-production-health/<timestamp>/
|
||||
```
|
||||
|
||||
巡检内容:
|
||||
|
||||
1. 通过 SSH 读取远端 Docker 容器状态。
|
||||
2. 检查后端 `8001` 和前端 `3003` 端口。
|
||||
3. 检查 `/health`。
|
||||
4. 检查 `/api/v1/system/readiness`。
|
||||
5. 检查 `/login` 和 `/pilot-readiness`。
|
||||
6. 验证默认管理员初始密码已被拒绝。
|
||||
|
||||
PASS 标准:
|
||||
|
||||
1. required 检查全部通过。
|
||||
2. readiness 为 `ready`。
|
||||
3. readiness `warnings=0`。
|
||||
4. readiness `password_rotated=true`。
|
||||
5. 默认管理员初始密码登录返回 401。
|
||||
|
||||
---
|
||||
|
||||
## 三、管理员密码口径
|
||||
|
||||
1. 管理员真实密码不得写入 Git、文档、脚本、`.env` 或日志。
|
||||
2. 自动化 E2E 需要管理员登录时,只通过当前终端环境变量提供:
|
||||
|
||||
```powershell
|
||||
$env:AI_CONTRACT_ADMIN_PASSWORD = '<由管理员输入的实际密码>'
|
||||
```
|
||||
|
||||
3. 默认初始密码只能用于验证“已经失效”,不能用于生产登录。
|
||||
4. 密码轮换脚本:
|
||||
|
||||
```powershell
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\rotate-admin-password.ps1 -ApiBase 'http://192.168.3.90:8001/api/v1'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 四、备份恢复口径
|
||||
|
||||
阶段 5 下一步将新增 `scripts/run-backup-drill.ps1`。当前原则先固定:
|
||||
|
||||
1. 生产备份可以读取生产数据。
|
||||
2. 恢复验证必须在隔离目录或临时数据库执行。
|
||||
3. 禁止把恢复演练直接覆盖生产库。
|
||||
4. 每次备份必须记录备份路径、文件大小、生成时间和校验方式。
|
||||
5. 每次恢复演练必须有独立日志和 PASS/FAIL 结论。
|
||||
|
||||
---
|
||||
|
||||
## 五、依赖治理口径
|
||||
|
||||
前端:
|
||||
|
||||
```powershell
|
||||
Push-Location frontend
|
||||
npm audit --json
|
||||
npm run lint
|
||||
npm run build
|
||||
Pop-Location
|
||||
```
|
||||
|
||||
后端:
|
||||
|
||||
```powershell
|
||||
backend\.venv\Scripts\python.exe -m pip check
|
||||
backend\.venv\Scripts\python.exe -m pytest -q backend/tests
|
||||
```
|
||||
|
||||
门禁:
|
||||
|
||||
1. high/critical 依赖漏洞必须为 0。
|
||||
2. moderate 漏洞必须修复或写明延期理由、影响范围、替代控制措施。
|
||||
3. 依赖升级后必须复跑构建、测试和关键 E2E。
|
||||
|
||||
---
|
||||
|
||||
## 六、常见故障处理
|
||||
|
||||
查看容器状态:
|
||||
|
||||
```bash
|
||||
cd /opt/ai-contract-platform
|
||||
docker compose -f docker-compose.server.yml ps
|
||||
```
|
||||
|
||||
查看后端日志:
|
||||
|
||||
```bash
|
||||
cd /opt/ai-contract-platform
|
||||
docker compose -f docker-compose.server.yml logs --tail=200 backend
|
||||
```
|
||||
|
||||
查看前端日志:
|
||||
|
||||
```bash
|
||||
cd /opt/ai-contract-platform
|
||||
docker compose -f docker-compose.server.yml logs --tail=200 frontend
|
||||
```
|
||||
|
||||
重启服务:
|
||||
|
||||
```bash
|
||||
cd /opt/ai-contract-platform
|
||||
docker compose -f docker-compose.server.yml up -d backend frontend
|
||||
```
|
||||
|
||||
重启后必须重新执行生产健康巡检。
|
||||
@@ -14,11 +14,11 @@
|
||||
| 阶段 2:深度开发与交付体验补强 | PASS | `project-docs/阶段2-深度开发与交付体验补强执行记录-2026-05-01.md` |
|
||||
| 阶段 3:试点交付准备 | PASS | `project-docs/阶段3-试点交付准备执行记录-2026-05-01.md` |
|
||||
| 阶段 4:试点运行监控与安全加固 | PASS | `project-docs/阶段4-试点运行监控与安全加固执行记录-2026-05-01.md` |
|
||||
| 阶段 5:生产运行、安全、监控、备份与依赖治理 | READY | `project-docs/阶段5-生产运行安全监控备份与依赖治理方案-2026-05-01.md` |
|
||||
| 阶段 5:生产运行、安全、监控、备份与依赖治理 | IN PROGRESS | `project-docs/阶段5-生产运行安全监控备份与依赖治理执行记录-2026-05-01.md` |
|
||||
|
||||
阶段 1 已完成源码闭环与生产入口闭环,生产无人 E2E `18/18` 通过。阶段 2 已完成 Review 重试、Knowledge Base 文档重试、Review -> Chat -> Contract 上下文联动、首页交付工作台和阶段 2 自动化无人闭环,本地与生产 E2E 均 `7/7` 通过。阶段 3 已完成系统 readiness、试点交付准备页、默认账号说明、演示样本数据、依赖健康门禁和阶段 3 自动化无人闭环,本地与生产 E2E 均 `5/5` 通过。
|
||||
|
||||
当前后续重点为“生产运行、安全、监控、备份与依赖治理”:默认密码轮换已完成,下一阶段优先建立生产健康巡检、备份恢复演练、依赖治理、安全基线和生产回归闭环。
|
||||
当前后续重点为“生产运行、安全、监控、备份与依赖治理”:默认密码轮换已完成,阶段 5 Step 1 生产健康巡检自动化已完成并 PASS,下一步进入备份恢复演练。
|
||||
|
||||
---
|
||||
|
||||
@@ -210,4 +210,4 @@ AI合同平台当前已经不是从零建设阶段。登录、合同审查、Rev
|
||||
|
||||
## 六、当前一句话结论
|
||||
|
||||
AI合同平台阶段 1、阶段 2 和阶段 3 均已完成闭环交付验证,阶段 4 已完成默认管理员密码轮换闭环,当前准备进入阶段 5:生产运行、安全、监控、备份与依赖治理。
|
||||
AI合同平台阶段 1、阶段 2 和阶段 3 均已完成闭环交付验证,阶段 4 已完成默认管理员密码轮换闭环,阶段 5 已启动并完成生产健康巡检自动化,下一步进入备份恢复演练。
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
# 阶段 5:生产运行、安全、监控、备份与依赖治理执行记录(2026-05-01)
|
||||
|
||||
> 当前目标:阶段 5 不再继续补主业务流程,而是建立生产运行、安全、监控、备份和依赖治理的可复跑闭环。
|
||||
|
||||
---
|
||||
|
||||
## 一、当前执行结论
|
||||
|
||||
| 主线 | 状态 | 备注 |
|
||||
|---|---|---|
|
||||
| gstack 安装校验 | PASS | 自动化执行前已返回 `GSTACK_OK` |
|
||||
| 代码提交与推送 | PASS | 已推送到 `origin/main`,远端为 `192.168.3.90` 仓库 |
|
||||
| Step 1:生产健康巡检自动化 | PASS | `scripts/run-production-health-check.ps1` 已实现并完成生产巡检 |
|
||||
| Step 2:备份与恢复演练 | PENDING | 下一步进入 |
|
||||
| Step 3:依赖治理专项 | PENDING | 待备份基线后处理 |
|
||||
| Step 4:安全基线检查 | PENDING | 将复用 Step 1 readiness 与默认密码验证 |
|
||||
| Step 5:生产回归闭环 | PENDING | 待运维加固动作后复跑阶段 1/2/3 |
|
||||
|
||||
阶段 5 Step 1 当前判定:PASS。
|
||||
|
||||
---
|
||||
|
||||
## 二、Step 1 生产健康巡检自动化
|
||||
|
||||
新增脚本:
|
||||
|
||||
```powershell
|
||||
scripts\run-production-health-check.ps1
|
||||
```
|
||||
|
||||
默认检查目标:
|
||||
|
||||
| 项目 | 默认值 |
|
||||
|---|---|
|
||||
| 服务器 | `192.168.3.90` |
|
||||
| 后端入口 | `http://192.168.3.90:8001` |
|
||||
| 前端入口 | `http://192.168.3.90:3003` |
|
||||
| 远端项目目录 | `/opt/ai-contract-platform` |
|
||||
| Compose 文件 | `docker-compose.server.yml` |
|
||||
|
||||
本次巡检产物:
|
||||
|
||||
```text
|
||||
project-docs/qa-artifacts/phase5-production-health/2026-05-01T08-32-50/
|
||||
```
|
||||
|
||||
说明:`qa-artifacts` 属于本地/服务器运行证据,已从 Git 版本化中排除,避免把截图、日志和压缩包推入源码仓库。
|
||||
|
||||
---
|
||||
|
||||
## 三、本次生产巡检结果
|
||||
|
||||
| 检查项 | 结果 | 证据 |
|
||||
|---|---|---|
|
||||
| Docker 容器状态 | PASS | backend/frontend 可读取且运行中 |
|
||||
| 后端端口 `8001` | PASS | TCP 可连接 |
|
||||
| 前端端口 `3003` | PASS | TCP 可连接 |
|
||||
| 后端 `/health` | PASS | HTTP 200 |
|
||||
| `/api/v1/system/readiness` | PASS | `ready`、`warnings=0`、`password_rotated=true` |
|
||||
| 前端 `/login` | PASS | HTTP 200 |
|
||||
| 前端 `/pilot-readiness` | PASS | HTTP 200 |
|
||||
| 默认管理员初始密码拒绝 | PASS | 登录返回 401 |
|
||||
|
||||
汇总文件:
|
||||
|
||||
```text
|
||||
production-health-summary.json
|
||||
production-health-summary.md
|
||||
container-status.txt
|
||||
health.json
|
||||
readiness.json
|
||||
default-password-login.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 四、脚本设计口径
|
||||
|
||||
1. 不读取、不写入管理员真实密码。
|
||||
2. 默认只验证初始密码已失效。
|
||||
3. SSH 只读取容器状态,不执行重启、删除、迁移等破坏性动作。
|
||||
4. 所有检查结果写入 JSON 和 Markdown,便于后续巡检、监控、告警和审计复用。
|
||||
5. 任何 required 检查失败时脚本退出失败。
|
||||
|
||||
---
|
||||
|
||||
## 五、下一步
|
||||
|
||||
阶段 5 下一步进入 Step 2:备份与恢复演练。
|
||||
|
||||
优先交付:
|
||||
|
||||
1. `scripts/run-backup-drill.ps1`
|
||||
2. PostgreSQL 备份文件生成和校验
|
||||
3. 部署配置、脚本、文档备份
|
||||
4. 隔离恢复可用性验证
|
||||
5. 备份恢复演练记录
|
||||
@@ -172,6 +172,13 @@ $env:AI_CONTRACT_ADMIN_PASSWORD = '<管理员实际密码>'
|
||||
2. `project-docs/阶段5-生产运行安全监控备份与依赖治理执行记录-2026-05-01.md`
|
||||
3. `project-docs/运维手册-2026-05-01.md`
|
||||
|
||||
当前执行状态:
|
||||
|
||||
- Step 1 已完成:`scripts/run-production-health-check.ps1`
|
||||
- 生产巡检结果:PASS
|
||||
- 巡检产物:`project-docs/qa-artifacts/phase5-production-health/2026-05-01T08-32-50/`
|
||||
- 下一步:Step 2 备份与恢复演练
|
||||
|
||||
---
|
||||
|
||||
## 七、一句话结论
|
||||
|
||||
@@ -0,0 +1,333 @@
|
||||
param(
|
||||
[string]$ServerHost = "192.168.3.90",
|
||||
[string]$BackendBaseUrl = "http://192.168.3.90:8001",
|
||||
[string]$FrontendBaseUrl = "http://192.168.3.90:3003",
|
||||
[string]$SshUser = "root",
|
||||
[string]$SshKeyPath = "$HOME\.ssh\water_audit_ed25519",
|
||||
[string]$RemoteProjectPath = "/opt/ai-contract-platform",
|
||||
[string]$ComposeFile = "docker-compose.server.yml",
|
||||
[string]$ArtifactRoot = "",
|
||||
[string]$AdminAccount = "admin@example.com",
|
||||
[string]$DefaultPassword = "123456",
|
||||
[switch]$SkipSsh
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
|
||||
$repoRoot = Resolve-Path (Join-Path $PSScriptRoot "..")
|
||||
$timestamp = Get-Date -Format "yyyy-MM-ddTHH-mm-ss"
|
||||
if ([string]::IsNullOrWhiteSpace($ArtifactRoot)) {
|
||||
$ArtifactRoot = Join-Path $repoRoot "project-docs\qa-artifacts\phase5-production-health"
|
||||
}
|
||||
$artifactDir = Join-Path $ArtifactRoot $timestamp
|
||||
$summaryJson = Join-Path $artifactDir "production-health-summary.json"
|
||||
$summaryMd = Join-Path $artifactDir "production-health-summary.md"
|
||||
$containerStatusPath = Join-Path $artifactDir "container-status.txt"
|
||||
$readinessPath = Join-Path $artifactDir "readiness.json"
|
||||
$healthPath = Join-Path $artifactDir "health.json"
|
||||
$defaultLoginPath = Join-Path $artifactDir "default-password-login.json"
|
||||
|
||||
New-Item -ItemType Directory -Force -Path $artifactDir | Out-Null
|
||||
|
||||
if (-not (Test-Path "$HOME\.Codex\skills\gstack\bin")) {
|
||||
throw "gstack is required for all AI-assisted work in this repo."
|
||||
}
|
||||
|
||||
Add-Type -AssemblyName System.Net.Http
|
||||
|
||||
$checks = New-Object System.Collections.Generic.List[object]
|
||||
|
||||
function Add-Check {
|
||||
param(
|
||||
[string]$Key,
|
||||
[string]$Label,
|
||||
[ValidateSet("pass", "fail", "warn")]
|
||||
[string]$Status,
|
||||
[string]$Detail,
|
||||
[bool]$Required = $true,
|
||||
[hashtable]$Meta = @{}
|
||||
)
|
||||
|
||||
$checks.Add([ordered]@{
|
||||
key = $Key
|
||||
label = $Label
|
||||
status = $Status
|
||||
required = $Required
|
||||
detail = $Detail
|
||||
meta = $Meta
|
||||
}) | Out-Null
|
||||
}
|
||||
|
||||
function ConvertTo-SafeFileName {
|
||||
param([string]$Value)
|
||||
return ($Value -replace '[^a-zA-Z0-9._-]', '-').Trim("-")
|
||||
}
|
||||
|
||||
function Save-Json {
|
||||
param(
|
||||
[string]$Path,
|
||||
[object]$Value
|
||||
)
|
||||
$Value | ConvertTo-Json -Depth 12 | Set-Content -Encoding UTF8 -Path $Path
|
||||
}
|
||||
|
||||
function Invoke-HttpProbe {
|
||||
param(
|
||||
[string]$Method,
|
||||
[string]$Url,
|
||||
[object]$Body = $null
|
||||
)
|
||||
|
||||
$handler = New-Object System.Net.Http.HttpClientHandler
|
||||
$client = New-Object System.Net.Http.HttpClient($handler)
|
||||
$client.Timeout = [TimeSpan]::FromSeconds(15)
|
||||
try {
|
||||
$httpMethod = New-Object System.Net.Http.HttpMethod($Method)
|
||||
$request = New-Object System.Net.Http.HttpRequestMessage($httpMethod, $Url)
|
||||
if ($null -ne $Body) {
|
||||
$json = $Body | ConvertTo-Json -Depth 8 -Compress
|
||||
$request.Content = New-Object System.Net.Http.StringContent($json, [System.Text.Encoding]::UTF8, "application/json")
|
||||
}
|
||||
|
||||
$response = $client.SendAsync($request).GetAwaiter().GetResult()
|
||||
$text = $response.Content.ReadAsStringAsync().GetAwaiter().GetResult()
|
||||
$parsed = $null
|
||||
if (-not [string]::IsNullOrWhiteSpace($text)) {
|
||||
try {
|
||||
$parsed = $text | ConvertFrom-Json -ErrorAction Stop
|
||||
} catch {
|
||||
$parsed = $null
|
||||
}
|
||||
}
|
||||
|
||||
return [ordered]@{
|
||||
ok = $true
|
||||
url = $Url
|
||||
method = $Method
|
||||
status_code = [int]$response.StatusCode
|
||||
body = $parsed
|
||||
body_text = if ($parsed) { $null } else { $text }
|
||||
}
|
||||
} catch {
|
||||
return [ordered]@{
|
||||
ok = $false
|
||||
url = $Url
|
||||
method = $Method
|
||||
status_code = $null
|
||||
error = $_.Exception.Message
|
||||
}
|
||||
} finally {
|
||||
$client.Dispose()
|
||||
$handler.Dispose()
|
||||
}
|
||||
}
|
||||
|
||||
function Test-TcpPort {
|
||||
param(
|
||||
[string]$HostName,
|
||||
[int]$Port,
|
||||
[int]$TimeoutMs = 3000
|
||||
)
|
||||
|
||||
$client = New-Object System.Net.Sockets.TcpClient
|
||||
try {
|
||||
$async = $client.BeginConnect($HostName, $Port, $null, $null)
|
||||
$connected = $async.AsyncWaitHandle.WaitOne($TimeoutMs, $false)
|
||||
if (-not $connected) {
|
||||
return $false
|
||||
}
|
||||
$client.EndConnect($async)
|
||||
return $client.Connected
|
||||
} catch {
|
||||
return $false
|
||||
} finally {
|
||||
$client.Close()
|
||||
}
|
||||
}
|
||||
|
||||
function Get-UriHostAndPort {
|
||||
param(
|
||||
[string]$Url,
|
||||
[int]$DefaultPort
|
||||
)
|
||||
$uri = [Uri]$Url
|
||||
$port = if ($uri.IsDefaultPort) { $DefaultPort } else { $uri.Port }
|
||||
return [ordered]@{
|
||||
host = $uri.Host
|
||||
port = $port
|
||||
}
|
||||
}
|
||||
|
||||
function Invoke-SshCommand {
|
||||
param([string]$Command)
|
||||
|
||||
$target = "${SshUser}@${ServerHost}"
|
||||
$args = @(
|
||||
"-i", $SshKeyPath,
|
||||
"-o", "BatchMode=yes",
|
||||
"-o", "StrictHostKeyChecking=accept-new",
|
||||
"-o", "ConnectTimeout=10",
|
||||
$target,
|
||||
$Command
|
||||
)
|
||||
$output = & ssh @args 2>&1
|
||||
return [ordered]@{
|
||||
exit_code = $LASTEXITCODE
|
||||
output = ($output -join [Environment]::NewLine)
|
||||
}
|
||||
}
|
||||
|
||||
$backendTarget = Get-UriHostAndPort -Url $BackendBaseUrl -DefaultPort 80
|
||||
$frontendTarget = Get-UriHostAndPort -Url $FrontendBaseUrl -DefaultPort 80
|
||||
|
||||
if ($SkipSsh) {
|
||||
Add-Check -Key "container_status" -Label "Docker containers" -Status "warn" -Required $false -Detail "Skipped by -SkipSsh"
|
||||
} elseif (-not (Test-Path $SshKeyPath)) {
|
||||
Add-Check -Key "container_status" -Label "Docker containers" -Status "fail" -Detail "SSH key not found: $SshKeyPath"
|
||||
} else {
|
||||
$remoteCommand = "cd $RemoteProjectPath; docker compose -f $ComposeFile ps"
|
||||
$sshResult = Invoke-SshCommand -Command $remoteCommand
|
||||
$sshResult.output | Set-Content -Encoding UTF8 -Path $containerStatusPath
|
||||
$hasBackend = $sshResult.output -match "backend"
|
||||
$hasFrontend = $sshResult.output -match "frontend"
|
||||
$looksRunning = $sshResult.output -match "running|Up|healthy"
|
||||
if ($sshResult.exit_code -eq 0 -and $hasBackend -and $hasFrontend -and $looksRunning) {
|
||||
Add-Check -Key "container_status" -Label "Docker containers" -Status "pass" -Detail "backend/frontend containers are visible and running" -Meta @{
|
||||
artifact = $containerStatusPath
|
||||
}
|
||||
} else {
|
||||
Add-Check -Key "container_status" -Label "Docker containers" -Status "fail" -Detail "Could not confirm backend/frontend running state, SSH exit=$($sshResult.exit_code)" -Meta @{
|
||||
artifact = $containerStatusPath
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
foreach ($target in @(
|
||||
@{ key = "backend_port"; label = "Backend port"; host = $backendTarget.host; port = $backendTarget.port },
|
||||
@{ key = "frontend_port"; label = "Frontend port"; host = $frontendTarget.host; port = $frontendTarget.port }
|
||||
)) {
|
||||
$portOpen = Test-TcpPort -HostName $target.host -Port $target.port
|
||||
if ($portOpen) {
|
||||
Add-Check -Key $target.key -Label $target.label -Status "pass" -Detail "$($target.host):$($target.port) accepts TCP connections"
|
||||
} else {
|
||||
Add-Check -Key $target.key -Label $target.label -Status "fail" -Detail "$($target.host):$($target.port) does not accept TCP connections"
|
||||
}
|
||||
}
|
||||
|
||||
$health = Invoke-HttpProbe -Method "GET" -Url "$BackendBaseUrl/health"
|
||||
Save-Json -Path $healthPath -Value $health
|
||||
if ($health.ok -and $health.status_code -eq 200) {
|
||||
Add-Check -Key "backend_health" -Label "Backend /health" -Status "pass" -Detail "/health returned 200" -Meta @{ artifact = $healthPath }
|
||||
} else {
|
||||
Add-Check -Key "backend_health" -Label "Backend /health" -Status "fail" -Detail "/health did not return 200" -Meta @{
|
||||
status_code = $health.status_code
|
||||
artifact = $healthPath
|
||||
}
|
||||
}
|
||||
|
||||
$readiness = Invoke-HttpProbe -Method "GET" -Url "$BackendBaseUrl/api/v1/system/readiness"
|
||||
Save-Json -Path $readinessPath -Value $readiness
|
||||
$readinessData = $readiness.body.data
|
||||
$passwordRotated = $false
|
||||
if ($readinessData -and $readinessData.pilot_assets -and $readinessData.pilot_assets.default_admin) {
|
||||
$passwordRotated = [bool]$readinessData.pilot_assets.default_admin.password_rotated
|
||||
}
|
||||
$warnings = if ($readinessData -and $readinessData.summary) { [int]$readinessData.summary.warnings } else { -1 }
|
||||
if ($readiness.ok -and $readiness.status_code -eq 200 -and $readiness.body.code -eq 0 -and $readinessData.status -eq "ready" -and $warnings -eq 0 -and $passwordRotated) {
|
||||
Add-Check -Key "system_readiness" -Label "System readiness" -Status "pass" -Detail "readiness=ready, warnings=0, password_rotated=true" -Meta @{ artifact = $readinessPath }
|
||||
} else {
|
||||
Add-Check -Key "system_readiness" -Label "System readiness" -Status "fail" -Detail "readiness did not satisfy ready/warnings=0/password_rotated=true" -Meta @{
|
||||
status_code = $readiness.status_code
|
||||
readiness_status = $readinessData.status
|
||||
warnings = $warnings
|
||||
password_rotated = $passwordRotated
|
||||
artifact = $readinessPath
|
||||
}
|
||||
}
|
||||
|
||||
foreach ($page in @("/login", "/pilot-readiness")) {
|
||||
$url = "$FrontendBaseUrl$page"
|
||||
$probe = Invoke-HttpProbe -Method "GET" -Url $url
|
||||
$fileName = "page-$((ConvertTo-SafeFileName -Value $page)).json"
|
||||
$pagePath = Join-Path $artifactDir $fileName
|
||||
Save-Json -Path $pagePath -Value $probe
|
||||
$key = "frontend_page_$((ConvertTo-SafeFileName -Value $page).ToLowerInvariant())"
|
||||
if ($probe.ok -and $probe.status_code -eq 200) {
|
||||
Add-Check -Key $key -Label "Frontend$page" -Status "pass" -Detail "$page returned 200" -Meta @{ artifact = $pagePath }
|
||||
} else {
|
||||
Add-Check -Key $key -Label "Frontend$page" -Status "fail" -Detail "$page did not return 200" -Meta @{
|
||||
status_code = $probe.status_code
|
||||
artifact = $pagePath
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
$defaultLogin = Invoke-HttpProbe -Method "POST" -Url "$BackendBaseUrl/api/v1/auth/login" -Body @{
|
||||
account = $AdminAccount
|
||||
password = $DefaultPassword
|
||||
}
|
||||
Save-Json -Path $defaultLoginPath -Value $defaultLogin
|
||||
if ($defaultLogin.ok -and $defaultLogin.status_code -eq 401) {
|
||||
Add-Check -Key "default_password_rejected" -Label "Default password rejection" -Status "pass" -Detail "Default admin password login returned 401" -Meta @{ artifact = $defaultLoginPath }
|
||||
} else {
|
||||
Add-Check -Key "default_password_rejected" -Label "Default password rejection" -Status "fail" -Detail "Default admin password was not clearly rejected" -Meta @{
|
||||
status_code = $defaultLogin.status_code
|
||||
artifact = $defaultLoginPath
|
||||
}
|
||||
}
|
||||
|
||||
$failedRequired = @($checks | Where-Object { $_.required -and $_.status -eq "fail" })
|
||||
$warningsList = @($checks | Where-Object { $_.status -eq "warn" })
|
||||
$overallStatus = if ($failedRequired.Count -gt 0) { "FAIL" } elseif ($warningsList.Count -gt 0) { "WARN" } else { "PASS" }
|
||||
|
||||
$summary = [ordered]@{
|
||||
name = "phase5-production-health"
|
||||
status = $overallStatus
|
||||
generated_at = (Get-Date).ToUniversalTime().ToString("o")
|
||||
artifact_dir = $artifactDir
|
||||
targets = [ordered]@{
|
||||
server_host = $ServerHost
|
||||
backend_base_url = $BackendBaseUrl
|
||||
frontend_base_url = $FrontendBaseUrl
|
||||
remote_project_path = $RemoteProjectPath
|
||||
compose_file = $ComposeFile
|
||||
}
|
||||
checks = $checks
|
||||
totals = [ordered]@{
|
||||
total = $checks.Count
|
||||
failed_required = $failedRequired.Count
|
||||
warnings = $warningsList.Count
|
||||
}
|
||||
}
|
||||
Save-Json -Path $summaryJson -Value $summary
|
||||
|
||||
$markdown = New-Object System.Collections.Generic.List[string]
|
||||
$markdown.Add("# Phase 5 Production Health Check") | Out-Null
|
||||
$markdown.Add("") | Out-Null
|
||||
$markdown.Add("- Status: $overallStatus") | Out-Null
|
||||
$markdown.Add("- Generated at: $($summary.generated_at)") | Out-Null
|
||||
$markdown.Add("- Backend: $BackendBaseUrl") | Out-Null
|
||||
$markdown.Add("- Frontend: $FrontendBaseUrl") | Out-Null
|
||||
$markdown.Add("- Artifact dir: $artifactDir") | Out-Null
|
||||
$markdown.Add("") | Out-Null
|
||||
$markdown.Add("| Check | Required | Status | Detail |") | Out-Null
|
||||
$markdown.Add("|---|---:|---|---|") | Out-Null
|
||||
foreach ($check in $checks) {
|
||||
$markdown.Add("| $($check.label) | $($check.required) | $($check.status.ToUpperInvariant()) | $($check.detail) |") | Out-Null
|
||||
}
|
||||
$markdown.Add("") | Out-Null
|
||||
$markdown.Add("Raw artifacts:") | Out-Null
|
||||
$markdown.Add("") | Out-Null
|
||||
$markdown.Add('- `production-health-summary.json`') | Out-Null
|
||||
$markdown.Add('- `readiness.json`') | Out-Null
|
||||
$markdown.Add('- `health.json`') | Out-Null
|
||||
$markdown.Add('- `default-password-login.json`') | Out-Null
|
||||
if (Test-Path $containerStatusPath) {
|
||||
$markdown.Add('- `container-status.txt`') | Out-Null
|
||||
}
|
||||
$markdown | Set-Content -Encoding UTF8 -Path $summaryMd
|
||||
|
||||
Write-Host "PHASE5_PRODUCTION_HEALTH_$overallStatus $artifactDir"
|
||||
if ($overallStatus -eq "FAIL") {
|
||||
throw "Phase 5 production health check failed. See $summaryMd"
|
||||
}
|
||||
Reference in New Issue
Block a user