diff --git a/project-docs/生产值守告警响应SOP-2026-05-01.md b/project-docs/生产值守告警响应SOP-2026-05-01.md new file mode 100644 index 00000000..3488eada --- /dev/null +++ b/project-docs/生产值守告警响应SOP-2026-05-01.md @@ -0,0 +1,80 @@ +# AI合同平台 生产值守告警响应 SOP（2026-05-01） + +> 适用范围：`AIContractProductionWatchdog` 计划任务、`run-production-watchdog-scheduler.ps1`、`run-production-watchdog.ps1` 产生的 `ok`、`warning`、`critical` 告警。 + +--- + +## 一、告警等级 + +| 等级 | 来源状态 | 处理口径 | +|---|---|---| +| `ok` | `PASS` | 记录测试消息即可，无需处理 | +| `warning` | `WARN` | 30 分钟内确认，查看 warning 检查项和子产物 | +| `critical` | `FAIL` | 10 分钟内确认，立即暂停发布并升级处理 | + +--- + +## 二、收到 `warning` 后 + +1. 打开告警中的 artifact 目录。 +2. 查看 `watchdog-scheduler-summary.md` 或 `production-watchdog-summary.md`。 +3. 找到 `WARN` 检查项。 +4. 如果是 dirty worktree，确认是否为正在开发的预期状态。 +5. 如果是磁盘、备份、Docker、健康接口、readiness 或页面访问告警，执行生产 ops check。 +6. 将处理结论写入响应记录。 + +命令： + +```powershell +powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-production-ops-check.ps1 +``` + +--- + +## 三、收到 `critical` 后 + +1. 暂停发布和回滚以外的变更。 +2. 打开告警 artifact 目录。 +3. 查看失败检查项和子产物。 +4. 查看远端容器状态和日志。 +5. 根据失败类型选择恢复动作：重启服务、容量治理、回滚 frontend、恢复备份或升级处理。 +6. 修复后重新执行 clean scheduler gate。 + +命令： + +```powershell +powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-production-watchdog-scheduler.ps1 -Mode RunOnce -RequireCleanWorktree -FailOnWarn -SkipWebhook +``` + +--- + +## 四、响应演练 + +读取最近的 scheduler 产物： + +```powershell +powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-watchdog-alert-response-drill.ps1 +``` + +读取指定 payload： + +```powershell +powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-watchdog-alert-response-drill.ps1 -AlertPayloadPath '' -ResponderName '<值守人>' +``` + +真实外部到人联调： + +```powershell +$env:AI_CONTRACT_WATCHDOG_WEBHOOK_URL = '<真实告警机器人或通知通道 URL>' +powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-production-watchdog-scheduler.ps1 -Mode RunOnce -RequireCleanWorktree -NotifyOnStatus All +Remove-Item Env:\AI_CONTRACT_WATCHDOG_WEBHOOK_URL +``` + +--- + +## 五、安全要求 + +1. 管理员密码不得写入脚本、文档、Git、计划任务或产物。 +2. Webhook URL 不得写入脚本、文档、Git、计划任务或产物。 +3. 外部告警到人必须由真实接收人确认。 +4. 本地 Webhook smoke 只能证明 HTTP 投递链路可用，不能证明外部人员已收到。 diff --git a/project-docs/运维手册-2026-05-01.md b/project-docs/运维手册-2026-05-01.md index bb6659f7..4d96e3ac 100644 --- a/project-docs/运维手册-2026-05-01.md +++ b/project-docs/运维手册-2026-05-01.md @@ -368,6 +368,26 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-production-w 真实外部告警联调前，必须在运行环境中设置 `AI_CONTRACT_WATCHDOG_WEBHOOK_URL`。该值不得进入 Git、脚本、文档或计划任务命令。 +告警响应 SOP： + +```text +project-docs/生产值守告警响应SOP-2026-05-01.md +``` + +执行告警响应演练： + +```powershell +powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-watchdog-alert-response-drill.ps1 +``` + +读取指定告警 payload 执行响应演练： + +```powershell +powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-watchdog-alert-response-drill.ps1 -AlertPayloadPath '' -ResponderName '<值守人>' +``` + +响应演练会生成 `alert-response-summary.json/md` 和 `alert-response-record.json/md`，用于证明收到告警后的确认、分级、处理路线和证据目录。 + 列出可回滚的 frontend 备份： ```powershell diff --git a/project-docs/闭环交付与深度开发方案-2026-05-01.md b/project-docs/闭环交付与深度开发方案-2026-05-01.md index eda0a26e..6377d214 100644 --- a/project-docs/闭环交付与深度开发方案-2026-05-01.md +++ b/project-docs/闭环交付与深度开发方案-2026-05-01.md @@ -16,11 +16,11 @@ | 阶段 4：试点运行监控与安全加固 | PASS | `project-docs/阶段4-试点运行监控与安全加固执行记录-2026-05-01.md` | | 阶段 5：生产运行、安全、监控、备份与依赖治理 | PASS | `project-docs/阶段5-生产运行安全监控备份与依赖治理执行记录-2026-05-01.md` | | 阶段 6：生产运营自动化与发布门禁固化 | PASS | `project-docs/阶段6-生产运营自动化与发布门禁固化执行记录-2026-05-01.md` | -| 阶段 7：生产运营常态化与一键发布 | PASS_WITH_WEBHOOK_BLOCKED | `project-docs/阶段7-一键前端Docker发布执行记录-2026-05-01.md`、`project-docs/阶段7.2-一键回滚与生产巡检门禁执行记录-2026-05-01.md`、`project-docs/阶段7.3-容量治理与备份保留策略执行记录-2026-05-01.md`、`project-docs/阶段7.4-日常巡检与告警门禁执行记录-2026-05-01.md`、`project-docs/阶段7.5-定时巡检与告警投递执行记录-2026-05-01.md`、`project-docs/阶段7.6-真实值守上线与告警联调执行记录-2026-05-01.md` | +| 阶段 7：生产运营常态化与一键发布 | PASS_WITH_WEBHOOK_BLOCKED | `project-docs/阶段7-一键前端Docker发布执行记录-2026-05-01.md`、`project-docs/阶段7.2-一键回滚与生产巡检门禁执行记录-2026-05-01.md`、`project-docs/阶段7.3-容量治理与备份保留策略执行记录-2026-05-01.md`、`project-docs/阶段7.4-日常巡检与告警门禁执行记录-2026-05-01.md`、`project-docs/阶段7.5-定时巡检与告警投递执行记录-2026-05-01.md`、`project-docs/阶段7.6-真实值守上线与告警联调执行记录-2026-05-01.md`、`project-docs/阶段7.7-告警到人与响应闭环执行记录-2026-05-01.md` | 阶段 1 已完成源码闭环与生产入口闭环，生产无人 E2E `18/18` 通过。阶段 2 已完成 Review 重试、Knowledge Base 文档重试、Review -> Chat -> Contract 上下文联动、首页交付工作台和阶段 2 自动化无人闭环，本地与生产 E2E 均 `7/7` 通过。阶段 3 已完成系统 readiness、试点交付准备页、默认账号说明、演示样本数据、依赖健康门禁和阶段 3 自动化无人闭环，本地与生产 E2E 均 `5/5` 通过。 -当前后续重点已从“生产运营自动化与发布门禁固化”推进到“生产运营常态化与一键发布”：阶段 7 首批一键发布闭环已完成，阶段 7.2 一键回滚与生产巡检门禁也已完成首批闭环。阶段 7.3 容量治理已完成，备份保留策略、Docker build cache/旧镜像清理和磁盘告警阈值已固化；服务器磁盘从 `89%` 降至 `67%`。阶段 7.4 日常巡检与告警门禁已完成，将 ops check、capacity governance dry run 和可选 release gate 聚合为统一生产值守入口，完整值守验证 `PHASE7_PRODUCTION_WATCHDOG_PASS`。阶段 7.5 已完成定时任务包装和 Webhook dry-run 告警投递，最终调度门禁 `PHASE7_WATCHDOG_SCHEDULER_PASS`。阶段 7.6 已完成真实计划任务上线和本地非 dry-run Webhook POST 烟测；当前外部告警到人仍需要运行时提供 `AI_CONTRACT_WATCHDOG_WEBHOOK_URL`。 +当前后续重点已从“生产运营自动化与发布门禁固化”推进到“生产运营常态化与一键发布”：阶段 7 首批一键发布闭环已完成，阶段 7.2 一键回滚与生产巡检门禁也已完成首批闭环。阶段 7.3 容量治理已完成，备份保留策略、Docker build cache/旧镜像清理和磁盘告警阈值已固化；服务器磁盘从 `89%` 降至 `67%`。阶段 7.4 日常巡检与告警门禁已完成，将 ops check、capacity governance dry run 和可选 release gate 聚合为统一生产值守入口，完整值守验证 `PHASE7_PRODUCTION_WATCHDOG_PASS`。阶段 7.5 已完成定时任务包装和 Webhook dry-run 告警投递，最终调度门禁 `PHASE7_WATCHDOG_SCHEDULER_PASS`。阶段 7.6 已完成真实计划任务上线和本地非 dry-run Webhook POST 烟测。阶段 7.7 正在把收到告警后的响应动作固化为 SOP 和可审计演练；当前外部告警到人仍需要运行时提供 `AI_CONTRACT_WATCHDOG_WEBHOOK_URL`。 --- @@ -220,4 +220,4 @@ AI合同平台当前已经不是从零建设阶段。登录、合同审查、Rev ## 六、当前一句话结论 -AI合同平台阶段 1、阶段 2、阶段 3、阶段 4、阶段 5、阶段 6、阶段 7、阶段 7.2、阶段 7.3、阶段 7.4 和阶段 7.5 均已完成闭环交付验证。阶段 7.6 已完成真实计划任务上线、本地真实 HTTP 告警投递烟测和 clean scheduler gate，但真实外部告警到人仍等待运行时 Webhook URL。当前系统具备主业务闭环、试点交付准备、生产健康巡检、备份恢复演练、依赖治理、安全基线、生产关键 E2E 回归、前端视觉刷新验收、统一发布门禁证据、Docker 一键发布、一键回滚、容量治理证据、日常生产值守告警入口、定时巡检包装和 Webhook 投递链路。阶段 7.3 已解除服务器磁盘容量 WARN，阶段 7.4 已将常态巡检统一固化为 `scripts/run-production-watchdog.ps1`，阶段 7.5/7.6 已将其包装为真实上线的 `AIContractProductionWatchdog` 计划任务。 +AI合同平台阶段 1、阶段 2、阶段 3、阶段 4、阶段 5、阶段 6、阶段 7、阶段 7.2、阶段 7.3、阶段 7.4 和阶段 7.5 均已完成闭环交付验证。阶段 7.6 已完成真实计划任务上线、本地真实 HTTP 告警投递烟测和 clean scheduler gate。阶段 7.7 正在补齐告警响应 SOP 与响应演练，但真实外部告警到人仍等待运行时 Webhook URL。当前系统具备主业务闭环、试点交付准备、生产健康巡检、备份恢复演练、依赖治理、安全基线、生产关键 E2E 回归、前端视觉刷新验收、统一发布门禁证据、Docker 一键发布、一键回滚、容量治理证据、日常生产值守告警入口、定时巡检包装、Webhook 投递链路和告警响应演练入口。阶段 7.3 已解除服务器磁盘容量 WARN，阶段 7.4 已将常态巡检统一固化为 `scripts/run-production-watchdog.ps1`，阶段 7.5/7.6 已将其包装为真实上线的 `AIContractProductionWatchdog` 计划任务。 diff --git a/project-docs/阶段7.7-告警到人与响应闭环执行记录-2026-05-01.md b/project-docs/阶段7.7-告警到人与响应闭环执行记录-2026-05-01.md new file mode 100644 index 00000000..b3efd5dc --- /dev/null +++ b/project-docs/阶段7.7-告警到人与响应闭环执行记录-2026-05-01.md @@ -0,0 +1,86 @@ +# 阶段 7.7：告警到人与响应闭环执行记录（2026-05-01） + +> 当前目标：完成“告警到人 + 人能响应”的生产值守闭环。真实外部到人需要运行时提供 `AI_CONTRACT_WATCHDOG_WEBHOOK_URL`；在 URL 缺失前，先完成响应 SOP 和响应演练自动化。 + +--- + +## 一、当前执行结论 + +| 主线 | 状态 | 备注 | +|---|---|---| +| gstack 安装校验 | PASS | 已返回 `GSTACK_OK` | +| 阶段 7.7 方案 | READY | 已新增阶段 7.7 方案文档 | +| 响应 SOP | READY | 已新增生产值守告警响应 SOP | +| 响应演练脚本 | READY | 已新增 `scripts/run-watchdog-alert-response-drill.ps1` | +| 响应演练 | WARN | 响应动作已记录；WARN 来自真实外部 Webhook URL 未配置 | +| clean scheduler gate | PENDING | 待执行 | +| 外部告警到人 | BLOCKED | 需要真实 `AI_CONTRACT_WATCHDOG_WEBHOOK_URL` | + +--- + +## 二、脚本能力 + +新增脚本： + +```powershell +scripts\run-watchdog-alert-response-drill.ps1 +``` + +输出产物： + +1. `alert-response-summary.json` +2. `alert-response-summary.md` +3. `alert-response-record.json` +4. `alert-response-record.md` + +--- + +## 三、响应演练 + +输入 payload： + +```text +project-docs\qa-artifacts\phase7.6-real-webhook-smoke\2026-05-01T13-40-43-497\received-webhook.json +``` + +命令： + +```powershell +powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-watchdog-alert-response-drill.ps1 -AlertPayloadPath 'D:\ai-contract-platform\project-docs\qa-artifacts\phase7.6-real-webhook-smoke\2026-05-01T13-40-43-497\received-webhook.json' -ResponderName 'AIContract-OnCall-Drill' -Action Acknowledge -ResponseNote 'Phase 7.7 drill: acknowledge warning alert and follow warning response SOP' +``` + +结果： + +```text +PHASE7_7_ALERT_RESPONSE_DRILL_WARN D:\ai-contract-platform\project-docs\qa-artifacts\phase7.7-alert-response-drill\2026-05-01T13-55-48-294 +``` + +顶层检查： + +| 检查项 | 结果 | 说明 | +|---|---|---| +| gstack preflight | PASS | `GSTACK_OK` | +| watchdog scheduled task | PASS | `AIContractProductionWatchdog state=Ready` | +| alert source | PASS | payload 可解析 | +| responder identity | PASS | 响应人已记录 | +| external webhook | WARN | `AI_CONTRACT_WATCHDOG_WEBHOOK_URL` 未配置 | +| ack SLA | PASS | 演练确认已按运行时记录 | + +响应记录： + +```text +Incident=watchdog-2026-05-01T13-55-48-294 +Responder=AIContract-OnCall-Drill +Alert status=WARN +Alert route=warning +``` + +响应路线： + +1. 打开 watchdog summary artifact。 +2. 识别 warning 检查项和子产物。 +3. 如 warning 来源不清，运行 production ops check。 +4. 决定继续观察、修复或升级。 +5. 将响应结论写入 `alert-response-record.md`。 + +结论：人能响应的 SOP 和审计产物已跑通。真实外部告警到人仍被 Webhook URL 缺失阻塞。 diff --git a/project-docs/阶段7.7-告警到人与响应闭环方案-2026-05-01.md b/project-docs/阶段7.7-告警到人与响应闭环方案-2026-05-01.md new file mode 100644 index 00000000..3e158a9b --- /dev/null +++ b/project-docs/阶段7.7-告警到人与响应闭环方案-2026-05-01.md @@ -0,0 +1,36 @@ +# 阶段 7.7：告警到人与响应闭环方案（2026-05-01） + +> 前置状态：阶段 7.6 已完成真实计划任务上线和本地非 dry-run Webhook POST 烟测，但外部告警到人仍被 `AI_CONTRACT_WATCHDOG_WEBHOOK_URL` 缺失阻塞。阶段 7.7 的目标是把“人收到告警后如何响应”固化为 SOP 和可审计演练，并在真实 Webhook 可用后完成到人确认。 + +--- + +## 一、阶段目标 + +1. 固化告警响应 SOP。 +2. 新增告警响应演练脚本，读取真实或演练 payload。 +3. 记录值守人、响应动作、分级路由和证据目录。 +4. 对 `ok`、`warning`、`critical` 三类告警给出响应动作。 +5. 保持外部 Webhook URL 运行时注入，不写入 Git、文档、脚本或计划任务。 +6. 在真实 URL 可用后执行 `NotifyOnStatus All` 到人验证。 + +--- + +## 二、交付范围 + +| 主线 | 目标 | 交付物 | +|---|---|---| +| A. 响应演练脚本 | 把人响应动作变成可审计产物 | `scripts/run-watchdog-alert-response-drill.ps1` | +| B. 响应 SOP | 明确收到告警后做什么 | `project-docs/生产值守告警响应SOP-2026-05-01.md` | +| C. 响应证据 | 记录 ack、分级、下一步动作 | `alert-response-record.json/md` | +| D. 外部到人 | 真实 Webhook URL 可用后执行 | `AI_CONTRACT_WATCHDOG_WEBHOOK_URL` 运行时注入 | + +--- + +## 三、验收标准 + +1. 响应演练脚本语法校验通过。 +2. 可读取 7.6 本地真实 POST payload。 +3. 可生成告警响应记录。 +4. 计划任务状态仍为 `Ready`。 +5. clean scheduler gate 仍 PASS。 +6. 若无真实 Webhook URL，外部到人保持 BLOCKED，不伪造结论。 diff --git a/scripts/run-watchdog-alert-response-drill.ps1 b/scripts/run-watchdog-alert-response-drill.ps1 new file mode 100644 index 00000000..80025c88 --- /dev/null +++ b/scripts/run-watchdog-alert-response-drill.ps1 @@ -0,0 +1,346 @@ +param( + [string]$AlertPayloadPath = "", + [string]$SchedulerArtifactDir = "", + [string]$ArtifactRoot = "", + [string]$ResponderName = "", + [ValidateSet("Drill", "Acknowledge", "Investigate", "Resolve", "Escalate")] + [string]$Action = "Drill", + [string]$ResponseNote = "Phase 7.7 watchdog alert response drill", + [string]$WebhookUrlEnvVar = "AI_CONTRACT_WATCHDOG_WEBHOOK_URL", + [switch]$RequireExternalWebhook, + [switch]$EvaluateHistoricalSla, + [int]$WarningAckSlaMinutes = 30, + [int]$CriticalAckSlaMinutes = 10, + [switch]$FailOnWarn +) + +$ErrorActionPreference = "Stop" + +$repoRoot = Resolve-Path (Join-Path $PSScriptRoot "..") +$timestamp = Get-Date -Format "yyyy-MM-ddTHH-mm-ss-fff" +if ([string]::IsNullOrWhiteSpace($ArtifactRoot)) { + $ArtifactRoot = Join-Path $repoRoot "project-docs\qa-artifacts\phase7.7-alert-response-drill" +} +if (-not [System.IO.Path]::IsPathRooted($ArtifactRoot)) { + $ArtifactRoot = Join-Path $repoRoot $ArtifactRoot +} +$artifactDir = Join-Path $ArtifactRoot $timestamp +New-Item -ItemType Directory -Force -Path $artifactDir | Out-Null + +$summaryJson = Join-Path $artifactDir "alert-response-summary.json" +$summaryMd = Join-Path $artifactDir "alert-response-summary.md" +$responseRecordJson = Join-Path $artifactDir "alert-response-record.json" +$responseRecordMd = Join-Path $artifactDir "alert-response-record.md" + +$checks = New-Object System.Collections.Generic.List[object] + +function Add-Check { + param( + [string]$Key, + [string]$Label, + [ValidateSet("pass", "fail", "warn", "skip")] + [string]$Status, + [string]$Detail, + [bool]$Required = $true, + [hashtable]$Meta = @{} + ) + + $checks.Add([ordered]@{ + key = $Key + label = $Label + status = $Status + required = $Required + detail = $Detail + meta = $Meta + }) | Out-Null +} + +function Save-Json { + param( + [string]$Path, + [object]$Value + ) + $Value | ConvertTo-Json -Depth 12 | Set-Content -Encoding UTF8 -Path $Path +} + +function Find-LatestSchedulerArtifact { + $root = Join-Path $repoRoot "project-docs\qa-artifacts\phase7.5-watchdog-scheduler" + if (-not (Test-Path -LiteralPath $root)) { + return "" + } + + $candidates = Get-ChildItem -LiteralPath $root -Directory | + Sort-Object LastWriteTime -Descending | + Where-Object { + (Test-Path -LiteralPath (Join-Path $_.FullName "watchdog-notification-payload.json")) -or + (Test-Path -LiteralPath (Join-Path $_.FullName "watchdog-scheduler-summary.json")) + } + + if ($candidates) { + return $candidates[0].FullName + } + return "" +} + +function Resolve-AlertSource { + if (-not [string]::IsNullOrWhiteSpace($AlertPayloadPath)) { + if (-not (Test-Path -LiteralPath $AlertPayloadPath)) { + throw "AlertPayloadPath not found: $AlertPayloadPath" + } + return @{ + type = "payload" + path = (Resolve-Path -LiteralPath $AlertPayloadPath).Path + artifact_dir = Split-Path -Parent (Resolve-Path -LiteralPath $AlertPayloadPath).Path + } + } + + if ([string]::IsNullOrWhiteSpace($SchedulerArtifactDir)) { + $SchedulerArtifactDir = Find-LatestSchedulerArtifact + } + if ([string]::IsNullOrWhiteSpace($SchedulerArtifactDir)) { + throw "No scheduler artifact was provided or found" + } + if (-not (Test-Path -LiteralPath $SchedulerArtifactDir)) { + throw "SchedulerArtifactDir not found: $SchedulerArtifactDir" + } + + $payloadPath = Join-Path $SchedulerArtifactDir "watchdog-notification-payload.json" + if (Test-Path -LiteralPath $payloadPath) { + return @{ + type = "payload" + path = (Resolve-Path -LiteralPath $payloadPath).Path + artifact_dir = (Resolve-Path -LiteralPath $SchedulerArtifactDir).Path + } + } + + $summaryPath = Join-Path $SchedulerArtifactDir "watchdog-scheduler-summary.json" + if (Test-Path -LiteralPath $summaryPath) { + return @{ + type = "summary" + path = (Resolve-Path -LiteralPath $summaryPath).Path + artifact_dir = (Resolve-Path -LiteralPath $SchedulerArtifactDir).Path + } + } + + throw "No alert payload or scheduler summary found in $SchedulerArtifactDir" +} + +function Get-ResponseRoute { + param( + [string]$Status, + [string]$Severity + ) + + if ($Status -eq "FAIL" -or $Severity -eq "critical") { + return [ordered]@{ + route = "critical" + ack_sla_minutes = $CriticalAckSlaMinutes + first_actions = @( + "Open alert-response-summary.md and the linked watchdog artifact", + "Run production ops check", + "Inspect backend/frontend container logs", + "Freeze deploy or rollback activity until cause is known", + "Escalate to administrator or service owner" + ) + } + } + + if ($Status -eq "WARN" -or $Severity -eq "warning") { + return [ordered]@{ + route = "warning" + ack_sla_minutes = $WarningAckSlaMinutes + first_actions = @( + "Open the watchdog summary artifact", + "Identify warning checks and child artifacts", + "Run production ops check if warning source is unclear", + "Decide whether to monitor, remediate, or escalate", + "Record response decision in alert-response-record.md" + ) + } + } + + return [ordered]@{ + route = "ok" + ack_sla_minutes = 0 + first_actions = @( + "Record receipt if this is a test message", + "No incident action required", + "Wait for the next scheduled watchdog run" + ) + } +} + +if (-not (Test-Path "$HOME\.Codex\skills\gstack\bin")) { + Add-Check -Key "gstack" -Label "gstack preflight" -Status "fail" -Detail "gstack is missing" +} else { + Add-Check -Key "gstack" -Label "gstack preflight" -Status "pass" -Detail "GSTACK_OK" +} + +$task = Get-ScheduledTask -TaskName "AIContractProductionWatchdog" -ErrorAction SilentlyContinue +if ($null -eq $task) { + Add-Check -Key "scheduled_task" -Label "watchdog scheduled task" -Status "fail" -Detail "AIContractProductionWatchdog is not registered" +} elseif ($task.State -eq "Ready" -or $task.State -eq "Running") { + Add-Check -Key "scheduled_task" -Label "watchdog scheduled task" -Status "pass" -Detail "AIContractProductionWatchdog state=$($task.State)" +} else { + Add-Check -Key "scheduled_task" -Label "watchdog scheduled task" -Status "warn" -Required $false -Detail "AIContractProductionWatchdog state=$($task.State)" +} + +$source = $null +$sourceData = $null +try { + $source = Resolve-AlertSource + $sourceData = Get-Content -LiteralPath $source.path -Raw -Encoding UTF8 | ConvertFrom-Json + Add-Check -Key "alert_source" -Label "alert source" -Status "pass" -Detail "Alert source parsed" -Meta @{ + type = $source.type + path = $source.path + artifact_dir = $source.artifact_dir + } +} catch { + Add-Check -Key "alert_source" -Label "alert source" -Status "fail" -Detail $_.Exception.Message +} + +if ([string]::IsNullOrWhiteSpace($ResponderName)) { + $ResponderName = [Environment]::GetEnvironmentVariable("AI_CONTRACT_ONCALL_NAME") +} +if ([string]::IsNullOrWhiteSpace($ResponderName)) { + $ResponderName = [Environment]::UserName +} +if ([string]::IsNullOrWhiteSpace($ResponderName)) { + Add-Check -Key "responder" -Label "responder identity" -Status "fail" -Detail "ResponderName or AI_CONTRACT_ONCALL_NAME is required" +} else { + Add-Check -Key "responder" -Label "responder identity" -Status "pass" -Detail "Responder identity recorded" +} + +$externalWebhook = [Environment]::GetEnvironmentVariable($WebhookUrlEnvVar) +if ([string]::IsNullOrWhiteSpace($externalWebhook)) { + if ($RequireExternalWebhook) { + Add-Check -Key "external_webhook" -Label "external webhook" -Status "fail" -Detail "$WebhookUrlEnvVar is not configured" + } else { + Add-Check -Key "external_webhook" -Label "external webhook" -Status "warn" -Required $false -Detail "$WebhookUrlEnvVar is not configured; external receipt remains blocked" + } +} else { + Add-Check -Key "external_webhook" -Label "external webhook" -Status "pass" -Detail "$WebhookUrlEnvVar is configured" +} + +$status = "UNKNOWN" +$severity = "unknown" +$generatedAt = $null +$alertArtifactDir = "" +if ($sourceData) { + if ($source.type -eq "payload") { + if ($sourceData.status) { $status = [string]$sourceData.status } + if ($sourceData.severity) { $severity = [string]$sourceData.severity } + if ($sourceData.generated_at) { $generatedAt = [datetime]$sourceData.generated_at } + if ($sourceData.artifact_dir) { $alertArtifactDir = [string]$sourceData.artifact_dir } + } else { + if ($sourceData.status) { $status = [string]$sourceData.status } + if ($sourceData.alert_severity) { $severity = [string]$sourceData.alert_severity } + if ($sourceData.generated_at) { $generatedAt = [datetime]$sourceData.generated_at } + if ($sourceData.artifact_dir) { $alertArtifactDir = [string]$sourceData.artifact_dir } + } +} + +$route = Get-ResponseRoute -Status $status -Severity $severity +$ackAt = Get-Date +$ackAgeMinutes = $null +if ($EvaluateHistoricalSla -and $generatedAt) { + $ackAgeMinutes = [math]::Round(($ackAt.ToUniversalTime() - $generatedAt.ToUniversalTime()).TotalMinutes, 2) + if ($route.ack_sla_minutes -gt 0 -and $ackAgeMinutes -gt $route.ack_sla_minutes) { + Add-Check -Key "ack_sla" -Label "ack SLA" -Status "warn" -Required $false -Detail "Historical ack age ${ackAgeMinutes}m exceeds SLA $($route.ack_sla_minutes)m" + } else { + Add-Check -Key "ack_sla" -Label "ack SLA" -Status "pass" -Detail "Historical ack age ${ackAgeMinutes}m is within SLA" + } +} else { + Add-Check -Key "ack_sla" -Label "ack SLA" -Status "pass" -Detail "Drill acknowledgment recorded at run time" +} + +$record = [ordered]@{ + incident_id = "watchdog-$timestamp" + action = $Action + responder = $ResponderName + response_note = $ResponseNote + acknowledged_at = $ackAt.ToUniversalTime().ToString("o") + source = $source + alert = [ordered]@{ + status = $status + severity = $severity + generated_at = if ($generatedAt) { $generatedAt.ToUniversalTime().ToString("o") } else { $null } + artifact_dir = $alertArtifactDir + } + route = $route + external_receipt = [ordered]@{ + webhook_env_var = $WebhookUrlEnvVar + configured = -not [string]::IsNullOrWhiteSpace($externalWebhook) + verified_to_person = $false + } +} +Save-Json -Path $responseRecordJson -Value $record + +$recordMd = New-Object System.Collections.Generic.List[string] +$recordMd.Add("# Watchdog Alert Response Record") | Out-Null +$recordMd.Add("") | Out-Null +$recordMd.Add("- Incident: $($record.incident_id)") | Out-Null +$recordMd.Add("- Action: $Action") | Out-Null +$recordMd.Add("- Responder: $ResponderName") | Out-Null +$recordMd.Add("- Alert status: $status") | Out-Null +$recordMd.Add("- Alert severity: $severity") | Out-Null +$recordMd.Add("- Alert artifact: $alertArtifactDir") | Out-Null +$recordMd.Add("- Response note: $ResponseNote") | Out-Null +$recordMd.Add("") | Out-Null +$recordMd.Add("First actions:") | Out-Null +foreach ($item in $route.first_actions) { + $recordMd.Add("- $item") | Out-Null +} +$recordMd | Set-Content -Encoding UTF8 -Path $responseRecordMd + +$failedRequired = @($checks | Where-Object { $_.required -and $_.status -eq "fail" }) +$warningsList = @($checks | Where-Object { $_.status -eq "warn" }) +$overallStatus = if ($failedRequired.Count -gt 0) { "FAIL" } elseif ($warningsList.Count -gt 0) { "WARN" } else { "PASS" } +$alertSeverity = if ($failedRequired.Count -gt 0) { "critical" } elseif ($warningsList.Count -gt 0) { "warning" } else { "ok" } + +$summary = [ordered]@{ + name = "phase7.7-alert-response-drill" + status = $overallStatus + alert_severity = $alertSeverity + generated_at = (Get-Date).ToUniversalTime().ToString("o") + artifact_dir = $artifactDir + record = $record + checks = $checks + totals = [ordered]@{ + total = $checks.Count + failed_required = $failedRequired.Count + warnings = $warningsList.Count + } +} +Save-Json -Path $summaryJson -Value $summary + +$markdown = New-Object System.Collections.Generic.List[string] +$markdown.Add("# Phase 7.7 Alert Response Drill") | Out-Null +$markdown.Add("") | Out-Null +$markdown.Add("- Status: $overallStatus") | Out-Null +$markdown.Add("- Alert severity: $alertSeverity") | Out-Null +$markdown.Add("- Generated at: $($summary.generated_at)") | Out-Null +$markdown.Add("- Artifact dir: $artifactDir") | Out-Null +$markdown.Add("- Incident: $($record.incident_id)") | Out-Null +$markdown.Add("- Responder: $ResponderName") | Out-Null +$markdown.Add("- Alert status: $status") | Out-Null +$markdown.Add("- Alert route: $($route.route)") | Out-Null +$markdown.Add("") | Out-Null +$markdown.Add("| Check | Required | Status | Detail |") | Out-Null +$markdown.Add("|---|---:|---|---|") | Out-Null +foreach ($check in $checks) { + $markdown.Add("| $($check.label) | $($check.required) | $($check.status.ToUpperInvariant()) | $($check.detail) |") | Out-Null +} +$markdown.Add("") | Out-Null +$markdown.Add("Local artifacts:") | Out-Null +$markdown.Add("") | Out-Null +$markdown.Add('- `alert-response-summary.json`') | Out-Null +$markdown.Add('- `alert-response-summary.md`') | Out-Null +$markdown.Add('- `alert-response-record.json`') | Out-Null +$markdown.Add('- `alert-response-record.md`') | Out-Null +$markdown | Set-Content -Encoding UTF8 -Path $summaryMd + +Write-Host "PHASE7_7_ALERT_RESPONSE_DRILL_$overallStatus $artifactDir" +if ($overallStatus -eq "FAIL" -or ($FailOnWarn -and $overallStatus -eq "WARN")) { + throw "Phase 7.7 alert response drill $overallStatus. See $summaryMd" +}