chore: add watchdog alert response drill
This commit is contained in:
Codex
@@ -0,0 +1,80 @@
|
||||
# AI合同平台 生产值守告警响应 SOP（2026-05-01）
|
||||
|
||||
> 适用范围：`AIContractProductionWatchdog` 计划任务、`run-production-watchdog-scheduler.ps1`、`run-production-watchdog.ps1` 产生的 `ok`、`warning`、`critical` 告警。
|
||||
|
||||
---
|
||||
|
||||
## 一、告警等级
|
||||
|
||||
| 等级 | 来源状态 | 处理口径 |
|
||||
|---|---|---|
|
||||
| `ok` | `PASS` | 记录测试消息即可，无需处理 |
|
||||
| `warning` | `WARN` | 30 分钟内确认，查看 warning 检查项和子产物 |
|
||||
| `critical` | `FAIL` | 10 分钟内确认，立即暂停发布并升级处理 |
|
||||
|
||||
---
|
||||
|
||||
## 二、收到 `warning` 后
|
||||
|
||||
1. 打开告警中的 artifact 目录。
|
||||
2. 查看 `watchdog-scheduler-summary.md` 或 `production-watchdog-summary.md`。
|
||||
3. 找到 `WARN` 检查项。
|
||||
4. 如果是 dirty worktree，确认是否为正在开发的预期状态。
|
||||
5. 如果是磁盘、备份、Docker、健康接口、readiness 或页面访问告警，执行生产 ops check。
|
||||
6. 将处理结论写入响应记录。
|
||||
|
||||
命令：
|
||||
|
||||
```powershell
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-production-ops-check.ps1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 三、收到 `critical` 后
|
||||
|
||||
1. 暂停发布和回滚以外的变更。
|
||||
2. 打开告警 artifact 目录。
|
||||
3. 查看失败检查项和子产物。
|
||||
4. 查看远端容器状态和日志。
|
||||
5. 根据失败类型选择恢复动作：重启服务、容量治理、回滚 frontend、恢复备份或升级处理。
|
||||
6. 修复后重新执行 clean scheduler gate。
|
||||
|
||||
命令：
|
||||
|
||||
```powershell
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-production-watchdog-scheduler.ps1 -Mode RunOnce -RequireCleanWorktree -FailOnWarn -SkipWebhook
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 四、响应演练
|
||||
|
||||
读取最近的 scheduler 产物：
|
||||
|
||||
```powershell
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-watchdog-alert-response-drill.ps1
|
||||
```
|
||||
|
||||
读取指定 payload：
|
||||
|
||||
```powershell
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-watchdog-alert-response-drill.ps1 -AlertPayloadPath '<payload.json>' -ResponderName '<值守人>'
|
||||
```
|
||||
|
||||
真实外部到人联调：
|
||||
|
||||
```powershell
|
||||
$env:AI_CONTRACT_WATCHDOG_WEBHOOK_URL = '<真实告警机器人或通知通道 URL>'
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-production-watchdog-scheduler.ps1 -Mode RunOnce -RequireCleanWorktree -NotifyOnStatus All
|
||||
Remove-Item Env:\AI_CONTRACT_WATCHDOG_WEBHOOK_URL
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 五、安全要求
|
||||
|
||||
1. 管理员密码不得写入脚本、文档、Git、计划任务或产物。
|
||||
2. Webhook URL 不得写入脚本、文档、Git、计划任务或产物。
|
||||
3. 外部告警到人必须由真实接收人确认。
|
||||
4. 本地 Webhook smoke 只能证明 HTTP 投递链路可用，不能证明外部人员已收到。
|
||||
@@ -368,6 +368,26 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-production-w
|
||||
|
||||
真实外部告警联调前，必须在运行环境中设置 `AI_CONTRACT_WATCHDOG_WEBHOOK_URL`。该值不得进入 Git、脚本、文档或计划任务命令。
|
||||
|
||||
告警响应 SOP：
|
||||
|
||||
```text
|
||||
project-docs/生产值守告警响应SOP-2026-05-01.md
|
||||
```
|
||||
|
||||
执行告警响应演练：
|
||||
|
||||
```powershell
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-watchdog-alert-response-drill.ps1
|
||||
```
|
||||
|
||||
读取指定告警 payload 执行响应演练：
|
||||
|
||||
```powershell
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-watchdog-alert-response-drill.ps1 -AlertPayloadPath '<payload.json>' -ResponderName '<值守人>'
|
||||
```
|
||||
|
||||
响应演练会生成 `alert-response-summary.json/md` 和 `alert-response-record.json/md`，用于证明收到告警后的确认、分级、处理路线和证据目录。
|
||||
|
||||
列出可回滚的 frontend 备份：
|
||||
|
||||
```powershell
|
||||
|
||||
@@ -16,11 +16,11 @@
|
||||
| 阶段 4：试点运行监控与安全加固 | PASS | `project-docs/阶段4-试点运行监控与安全加固执行记录-2026-05-01.md` |
|
||||
| 阶段 5：生产运行、安全、监控、备份与依赖治理 | PASS | `project-docs/阶段5-生产运行安全监控备份与依赖治理执行记录-2026-05-01.md` |
|
||||
| 阶段 6：生产运营自动化与发布门禁固化 | PASS | `project-docs/阶段6-生产运营自动化与发布门禁固化执行记录-2026-05-01.md` |
|
||||
| 阶段 7：生产运营常态化与一键发布 | PASS_WITH_WEBHOOK_BLOCKED | `project-docs/阶段7-一键前端Docker发布执行记录-2026-05-01.md`、`project-docs/阶段7.2-一键回滚与生产巡检门禁执行记录-2026-05-01.md`、`project-docs/阶段7.3-容量治理与备份保留策略执行记录-2026-05-01.md`、`project-docs/阶段7.4-日常巡检与告警门禁执行记录-2026-05-01.md`、`project-docs/阶段7.5-定时巡检与告警投递执行记录-2026-05-01.md`、`project-docs/阶段7.6-真实值守上线与告警联调执行记录-2026-05-01.md` |
|
||||
| 阶段 7：生产运营常态化与一键发布 | PASS_WITH_WEBHOOK_BLOCKED | `project-docs/阶段7-一键前端Docker发布执行记录-2026-05-01.md`、`project-docs/阶段7.2-一键回滚与生产巡检门禁执行记录-2026-05-01.md`、`project-docs/阶段7.3-容量治理与备份保留策略执行记录-2026-05-01.md`、`project-docs/阶段7.4-日常巡检与告警门禁执行记录-2026-05-01.md`、`project-docs/阶段7.5-定时巡检与告警投递执行记录-2026-05-01.md`、`project-docs/阶段7.6-真实值守上线与告警联调执行记录-2026-05-01.md`、`project-docs/阶段7.7-告警到人与响应闭环执行记录-2026-05-01.md` |
|
||||
|
||||
阶段 1 已完成源码闭环与生产入口闭环，生产无人 E2E `18/18` 通过。阶段 2 已完成 Review 重试、Knowledge Base 文档重试、Review -> Chat -> Contract 上下文联动、首页交付工作台和阶段 2 自动化无人闭环，本地与生产 E2E 均 `7/7` 通过。阶段 3 已完成系统 readiness、试点交付准备页、默认账号说明、演示样本数据、依赖健康门禁和阶段 3 自动化无人闭环，本地与生产 E2E 均 `5/5` 通过。
|
||||
|
||||
当前后续重点已从“生产运营自动化与发布门禁固化”推进到“生产运营常态化与一键发布”：阶段 7 首批一键发布闭环已完成，阶段 7.2 一键回滚与生产巡检门禁也已完成首批闭环。阶段 7.3 容量治理已完成，备份保留策略、Docker build cache/旧镜像清理和磁盘告警阈值已固化；服务器磁盘从 `89%` 降至 `67%`。阶段 7.4 日常巡检与告警门禁已完成，将 ops check、capacity governance dry run 和可选 release gate 聚合为统一生产值守入口，完整值守验证 `PHASE7_PRODUCTION_WATCHDOG_PASS`。阶段 7.5 已完成定时任务包装和 Webhook dry-run 告警投递，最终调度门禁 `PHASE7_WATCHDOG_SCHEDULER_PASS`。阶段 7.6 已完成真实计划任务上线和本地非 dry-run Webhook POST 烟测；当前外部告警到人仍需要运行时提供 `AI_CONTRACT_WATCHDOG_WEBHOOK_URL`。
|
||||
当前后续重点已从“生产运营自动化与发布门禁固化”推进到“生产运营常态化与一键发布”：阶段 7 首批一键发布闭环已完成，阶段 7.2 一键回滚与生产巡检门禁也已完成首批闭环。阶段 7.3 容量治理已完成，备份保留策略、Docker build cache/旧镜像清理和磁盘告警阈值已固化；服务器磁盘从 `89%` 降至 `67%`。阶段 7.4 日常巡检与告警门禁已完成，将 ops check、capacity governance dry run 和可选 release gate 聚合为统一生产值守入口，完整值守验证 `PHASE7_PRODUCTION_WATCHDOG_PASS`。阶段 7.5 已完成定时任务包装和 Webhook dry-run 告警投递，最终调度门禁 `PHASE7_WATCHDOG_SCHEDULER_PASS`。阶段 7.6 已完成真实计划任务上线和本地非 dry-run Webhook POST 烟测。阶段 7.7 正在把收到告警后的响应动作固化为 SOP 和可审计演练；当前外部告警到人仍需要运行时提供 `AI_CONTRACT_WATCHDOG_WEBHOOK_URL`。
|
||||
|
||||
---
|
||||
|
||||
@@ -220,4 +220,4 @@ AI合同平台当前已经不是从零建设阶段。登录、合同审查、Rev
|
||||
|
||||
## 六、当前一句话结论
|
||||
|
||||
AI合同平台阶段 1、阶段 2、阶段 3、阶段 4、阶段 5、阶段 6、阶段 7、阶段 7.2、阶段 7.3、阶段 7.4 和阶段 7.5 均已完成闭环交付验证。阶段 7.6 已完成真实计划任务上线、本地真实 HTTP 告警投递烟测和 clean scheduler gate，但真实外部告警到人仍等待运行时 Webhook URL。当前系统具备主业务闭环、试点交付准备、生产健康巡检、备份恢复演练、依赖治理、安全基线、生产关键 E2E 回归、前端视觉刷新验收、统一发布门禁证据、Docker 一键发布、一键回滚、容量治理证据、日常生产值守告警入口、定时巡检包装和 Webhook 投递链路。阶段 7.3 已解除服务器磁盘容量 WARN，阶段 7.4 已将常态巡检统一固化为 `scripts/run-production-watchdog.ps1`，阶段 7.5/7.6 已将其包装为真实上线的 `AIContractProductionWatchdog` 计划任务。
|
||||
AI合同平台阶段 1、阶段 2、阶段 3、阶段 4、阶段 5、阶段 6、阶段 7、阶段 7.2、阶段 7.3、阶段 7.4 和阶段 7.5 均已完成闭环交付验证。阶段 7.6 已完成真实计划任务上线、本地真实 HTTP 告警投递烟测和 clean scheduler gate。阶段 7.7 正在补齐告警响应 SOP 与响应演练，但真实外部告警到人仍等待运行时 Webhook URL。当前系统具备主业务闭环、试点交付准备、生产健康巡检、备份恢复演练、依赖治理、安全基线、生产关键 E2E 回归、前端视觉刷新验收、统一发布门禁证据、Docker 一键发布、一键回滚、容量治理证据、日常生产值守告警入口、定时巡检包装、Webhook 投递链路和告警响应演练入口。阶段 7.3 已解除服务器磁盘容量 WARN，阶段 7.4 已将常态巡检统一固化为 `scripts/run-production-watchdog.ps1`，阶段 7.5/7.6 已将其包装为真实上线的 `AIContractProductionWatchdog` 计划任务。
|
||||
|
||||
@@ -0,0 +1,86 @@
|
||||
# 阶段 7.7：告警到人与响应闭环执行记录（2026-05-01）
|
||||
|
||||
> 当前目标：完成“告警到人 + 人能响应”的生产值守闭环。真实外部到人需要运行时提供 `AI_CONTRACT_WATCHDOG_WEBHOOK_URL`；在 URL 缺失前，先完成响应 SOP 和响应演练自动化。
|
||||
|
||||
---
|
||||
|
||||
## 一、当前执行结论
|
||||
|
||||
| 主线 | 状态 | 备注 |
|
||||
|---|---|---|
|
||||
| gstack 安装校验 | PASS | 已返回 `GSTACK_OK` |
|
||||
| 阶段 7.7 方案 | READY | 已新增阶段 7.7 方案文档 |
|
||||
| 响应 SOP | READY | 已新增生产值守告警响应 SOP |
|
||||
| 响应演练脚本 | READY | 已新增 `scripts/run-watchdog-alert-response-drill.ps1` |
|
||||
| 响应演练 | WARN | 响应动作已记录；WARN 来自真实外部 Webhook URL 未配置 |
|
||||
| clean scheduler gate | PENDING | 待执行 |
|
||||
| 外部告警到人 | BLOCKED | 需要真实 `AI_CONTRACT_WATCHDOG_WEBHOOK_URL` |
|
||||
|
||||
---
|
||||
|
||||
## 二、脚本能力
|
||||
|
||||
新增脚本：
|
||||
|
||||
```powershell
|
||||
scripts\run-watchdog-alert-response-drill.ps1
|
||||
```
|
||||
|
||||
输出产物：
|
||||
|
||||
1. `alert-response-summary.json`
|
||||
2. `alert-response-summary.md`
|
||||
3. `alert-response-record.json`
|
||||
4. `alert-response-record.md`
|
||||
|
||||
---
|
||||
|
||||
## 三、响应演练
|
||||
|
||||
输入 payload：
|
||||
|
||||
```text
|
||||
project-docs\qa-artifacts\phase7.6-real-webhook-smoke\2026-05-01T13-40-43-497\received-webhook.json
|
||||
```
|
||||
|
||||
命令：
|
||||
|
||||
```powershell
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-watchdog-alert-response-drill.ps1 -AlertPayloadPath 'D:\ai-contract-platform\project-docs\qa-artifacts\phase7.6-real-webhook-smoke\2026-05-01T13-40-43-497\received-webhook.json' -ResponderName 'AIContract-OnCall-Drill' -Action Acknowledge -ResponseNote 'Phase 7.7 drill: acknowledge warning alert and follow warning response SOP'
|
||||
```
|
||||
|
||||
结果：
|
||||
|
||||
```text
|
||||
PHASE7_7_ALERT_RESPONSE_DRILL_WARN D:\ai-contract-platform\project-docs\qa-artifacts\phase7.7-alert-response-drill\2026-05-01T13-55-48-294
|
||||
```
|
||||
|
||||
顶层检查：
|
||||
|
||||
| 检查项 | 结果 | 说明 |
|
||||
|---|---|---|
|
||||
| gstack preflight | PASS | `GSTACK_OK` |
|
||||
| watchdog scheduled task | PASS | `AIContractProductionWatchdog state=Ready` |
|
||||
| alert source | PASS | payload 可解析 |
|
||||
| responder identity | PASS | 响应人已记录 |
|
||||
| external webhook | WARN | `AI_CONTRACT_WATCHDOG_WEBHOOK_URL` 未配置 |
|
||||
| ack SLA | PASS | 演练确认已按运行时记录 |
|
||||
|
||||
响应记录：
|
||||
|
||||
```text
|
||||
Incident=watchdog-2026-05-01T13-55-48-294
|
||||
Responder=AIContract-OnCall-Drill
|
||||
Alert status=WARN
|
||||
Alert route=warning
|
||||
```
|
||||
|
||||
响应路线：
|
||||
|
||||
1. 打开 watchdog summary artifact。
|
||||
2. 识别 warning 检查项和子产物。
|
||||
3. 如 warning 来源不清，运行 production ops check。
|
||||
4. 决定继续观察、修复或升级。
|
||||
5. 将响应结论写入 `alert-response-record.md`。
|
||||
|
||||
结论：人能响应的 SOP 和审计产物已跑通。真实外部告警到人仍被 Webhook URL 缺失阻塞。
|
||||
@@ -0,0 +1,36 @@
|
||||
# 阶段 7.7：告警到人与响应闭环方案（2026-05-01）
|
||||
|
||||
> 前置状态：阶段 7.6 已完成真实计划任务上线和本地非 dry-run Webhook POST 烟测，但外部告警到人仍被 `AI_CONTRACT_WATCHDOG_WEBHOOK_URL` 缺失阻塞。阶段 7.7 的目标是把“人收到告警后如何响应”固化为 SOP 和可审计演练，并在真实 Webhook 可用后完成到人确认。
|
||||
|
||||
---
|
||||
|
||||
## 一、阶段目标
|
||||
|
||||
1. 固化告警响应 SOP。
|
||||
2. 新增告警响应演练脚本，读取真实或演练 payload。
|
||||
3. 记录值守人、响应动作、分级路由和证据目录。
|
||||
4. 对 `ok`、`warning`、`critical` 三类告警给出响应动作。
|
||||
5. 保持外部 Webhook URL 运行时注入，不写入 Git、文档、脚本或计划任务。
|
||||
6. 在真实 URL 可用后执行 `NotifyOnStatus All` 到人验证。
|
||||
|
||||
---
|
||||
|
||||
## 二、交付范围
|
||||
|
||||
| 主线 | 目标 | 交付物 |
|
||||
|---|---|---|
|
||||
| A. 响应演练脚本 | 把人响应动作变成可审计产物 | `scripts/run-watchdog-alert-response-drill.ps1` |
|
||||
| B. 响应 SOP | 明确收到告警后做什么 | `project-docs/生产值守告警响应SOP-2026-05-01.md` |
|
||||
| C. 响应证据 | 记录 ack、分级、下一步动作 | `alert-response-record.json/md` |
|
||||
| D. 外部到人 | 真实 Webhook URL 可用后执行 | `AI_CONTRACT_WATCHDOG_WEBHOOK_URL` 运行时注入 |
|
||||
|
||||
---
|
||||
|
||||
## 三、验收标准
|
||||
|
||||
1. 响应演练脚本语法校验通过。
|
||||
2. 可读取 7.6 本地真实 POST payload。
|
||||
3. 可生成告警响应记录。
|
||||
4. 计划任务状态仍为 `Ready`。
|
||||
5. clean scheduler gate 仍 PASS。
|
||||
6. 若无真实 Webhook URL，外部到人保持 BLOCKED，不伪造结论。
|
||||
@@ -0,0 +1,346 @@
|
||||
param(
|
||||
[string]$AlertPayloadPath = "",
|
||||
[string]$SchedulerArtifactDir = "",
|
||||
[string]$ArtifactRoot = "",
|
||||
[string]$ResponderName = "",
|
||||
[ValidateSet("Drill", "Acknowledge", "Investigate", "Resolve", "Escalate")]
|
||||
[string]$Action = "Drill",
|
||||
[string]$ResponseNote = "Phase 7.7 watchdog alert response drill",
|
||||
[string]$WebhookUrlEnvVar = "AI_CONTRACT_WATCHDOG_WEBHOOK_URL",
|
||||
[switch]$RequireExternalWebhook,
|
||||
[switch]$EvaluateHistoricalSla,
|
||||
[int]$WarningAckSlaMinutes = 30,
|
||||
[int]$CriticalAckSlaMinutes = 10,
|
||||
[switch]$FailOnWarn
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
|
||||
$repoRoot = Resolve-Path (Join-Path $PSScriptRoot "..")
|
||||
$timestamp = Get-Date -Format "yyyy-MM-ddTHH-mm-ss-fff"
|
||||
if ([string]::IsNullOrWhiteSpace($ArtifactRoot)) {
|
||||
$ArtifactRoot = Join-Path $repoRoot "project-docs\qa-artifacts\phase7.7-alert-response-drill"
|
||||
}
|
||||
if (-not [System.IO.Path]::IsPathRooted($ArtifactRoot)) {
|
||||
$ArtifactRoot = Join-Path $repoRoot $ArtifactRoot
|
||||
}
|
||||
$artifactDir = Join-Path $ArtifactRoot $timestamp
|
||||
New-Item -ItemType Directory -Force -Path $artifactDir | Out-Null
|
||||
|
||||
$summaryJson = Join-Path $artifactDir "alert-response-summary.json"
|
||||
$summaryMd = Join-Path $artifactDir "alert-response-summary.md"
|
||||
$responseRecordJson = Join-Path $artifactDir "alert-response-record.json"
|
||||
$responseRecordMd = Join-Path $artifactDir "alert-response-record.md"
|
||||
|
||||
$checks = New-Object System.Collections.Generic.List[object]
|
||||
|
||||
function Add-Check {
|
||||
param(
|
||||
[string]$Key,
|
||||
[string]$Label,
|
||||
[ValidateSet("pass", "fail", "warn", "skip")]
|
||||
[string]$Status,
|
||||
[string]$Detail,
|
||||
[bool]$Required = $true,
|
||||
[hashtable]$Meta = @{}
|
||||
)
|
||||
|
||||
$checks.Add([ordered]@{
|
||||
key = $Key
|
||||
label = $Label
|
||||
status = $Status
|
||||
required = $Required
|
||||
detail = $Detail
|
||||
meta = $Meta
|
||||
}) | Out-Null
|
||||
}
|
||||
|
||||
function Save-Json {
|
||||
param(
|
||||
[string]$Path,
|
||||
[object]$Value
|
||||
)
|
||||
$Value | ConvertTo-Json -Depth 12 | Set-Content -Encoding UTF8 -Path $Path
|
||||
}
|
||||
|
||||
function Find-LatestSchedulerArtifact {
|
||||
$root = Join-Path $repoRoot "project-docs\qa-artifacts\phase7.5-watchdog-scheduler"
|
||||
if (-not (Test-Path -LiteralPath $root)) {
|
||||
return ""
|
||||
}
|
||||
|
||||
$candidates = Get-ChildItem -LiteralPath $root -Directory |
|
||||
Sort-Object LastWriteTime -Descending |
|
||||
Where-Object {
|
||||
(Test-Path -LiteralPath (Join-Path $_.FullName "watchdog-notification-payload.json")) -or
|
||||
(Test-Path -LiteralPath (Join-Path $_.FullName "watchdog-scheduler-summary.json"))
|
||||
}
|
||||
|
||||
if ($candidates) {
|
||||
return $candidates[0].FullName
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
function Resolve-AlertSource {
|
||||
if (-not [string]::IsNullOrWhiteSpace($AlertPayloadPath)) {
|
||||
if (-not (Test-Path -LiteralPath $AlertPayloadPath)) {
|
||||
throw "AlertPayloadPath not found: $AlertPayloadPath"
|
||||
}
|
||||
return @{
|
||||
type = "payload"
|
||||
path = (Resolve-Path -LiteralPath $AlertPayloadPath).Path
|
||||
artifact_dir = Split-Path -Parent (Resolve-Path -LiteralPath $AlertPayloadPath).Path
|
||||
}
|
||||
}
|
||||
|
||||
if ([string]::IsNullOrWhiteSpace($SchedulerArtifactDir)) {
|
||||
$SchedulerArtifactDir = Find-LatestSchedulerArtifact
|
||||
}
|
||||
if ([string]::IsNullOrWhiteSpace($SchedulerArtifactDir)) {
|
||||
throw "No scheduler artifact was provided or found"
|
||||
}
|
||||
if (-not (Test-Path -LiteralPath $SchedulerArtifactDir)) {
|
||||
throw "SchedulerArtifactDir not found: $SchedulerArtifactDir"
|
||||
}
|
||||
|
||||
$payloadPath = Join-Path $SchedulerArtifactDir "watchdog-notification-payload.json"
|
||||
if (Test-Path -LiteralPath $payloadPath) {
|
||||
return @{
|
||||
type = "payload"
|
||||
path = (Resolve-Path -LiteralPath $payloadPath).Path
|
||||
artifact_dir = (Resolve-Path -LiteralPath $SchedulerArtifactDir).Path
|
||||
}
|
||||
}
|
||||
|
||||
$summaryPath = Join-Path $SchedulerArtifactDir "watchdog-scheduler-summary.json"
|
||||
if (Test-Path -LiteralPath $summaryPath) {
|
||||
return @{
|
||||
type = "summary"
|
||||
path = (Resolve-Path -LiteralPath $summaryPath).Path
|
||||
artifact_dir = (Resolve-Path -LiteralPath $SchedulerArtifactDir).Path
|
||||
}
|
||||
}
|
||||
|
||||
throw "No alert payload or scheduler summary found in $SchedulerArtifactDir"
|
||||
}
|
||||
|
||||
function Get-ResponseRoute {
|
||||
param(
|
||||
[string]$Status,
|
||||
[string]$Severity
|
||||
)
|
||||
|
||||
if ($Status -eq "FAIL" -or $Severity -eq "critical") {
|
||||
return [ordered]@{
|
||||
route = "critical"
|
||||
ack_sla_minutes = $CriticalAckSlaMinutes
|
||||
first_actions = @(
|
||||
"Open alert-response-summary.md and the linked watchdog artifact",
|
||||
"Run production ops check",
|
||||
"Inspect backend/frontend container logs",
|
||||
"Freeze deploy or rollback activity until cause is known",
|
||||
"Escalate to administrator or service owner"
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
if ($Status -eq "WARN" -or $Severity -eq "warning") {
|
||||
return [ordered]@{
|
||||
route = "warning"
|
||||
ack_sla_minutes = $WarningAckSlaMinutes
|
||||
first_actions = @(
|
||||
"Open the watchdog summary artifact",
|
||||
"Identify warning checks and child artifacts",
|
||||
"Run production ops check if warning source is unclear",
|
||||
"Decide whether to monitor, remediate, or escalate",
|
||||
"Record response decision in alert-response-record.md"
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
return [ordered]@{
|
||||
route = "ok"
|
||||
ack_sla_minutes = 0
|
||||
first_actions = @(
|
||||
"Record receipt if this is a test message",
|
||||
"No incident action required",
|
||||
"Wait for the next scheduled watchdog run"
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
if (-not (Test-Path "$HOME\.Codex\skills\gstack\bin")) {
|
||||
Add-Check -Key "gstack" -Label "gstack preflight" -Status "fail" -Detail "gstack is missing"
|
||||
} else {
|
||||
Add-Check -Key "gstack" -Label "gstack preflight" -Status "pass" -Detail "GSTACK_OK"
|
||||
}
|
||||
|
||||
$task = Get-ScheduledTask -TaskName "AIContractProductionWatchdog" -ErrorAction SilentlyContinue
|
||||
if ($null -eq $task) {
|
||||
Add-Check -Key "scheduled_task" -Label "watchdog scheduled task" -Status "fail" -Detail "AIContractProductionWatchdog is not registered"
|
||||
} elseif ($task.State -eq "Ready" -or $task.State -eq "Running") {
|
||||
Add-Check -Key "scheduled_task" -Label "watchdog scheduled task" -Status "pass" -Detail "AIContractProductionWatchdog state=$($task.State)"
|
||||
} else {
|
||||
Add-Check -Key "scheduled_task" -Label "watchdog scheduled task" -Status "warn" -Required $false -Detail "AIContractProductionWatchdog state=$($task.State)"
|
||||
}
|
||||
|
||||
$source = $null
|
||||
$sourceData = $null
|
||||
try {
|
||||
$source = Resolve-AlertSource
|
||||
$sourceData = Get-Content -LiteralPath $source.path -Raw -Encoding UTF8 | ConvertFrom-Json
|
||||
Add-Check -Key "alert_source" -Label "alert source" -Status "pass" -Detail "Alert source parsed" -Meta @{
|
||||
type = $source.type
|
||||
path = $source.path
|
||||
artifact_dir = $source.artifact_dir
|
||||
}
|
||||
} catch {
|
||||
Add-Check -Key "alert_source" -Label "alert source" -Status "fail" -Detail $_.Exception.Message
|
||||
}
|
||||
|
||||
if ([string]::IsNullOrWhiteSpace($ResponderName)) {
|
||||
$ResponderName = [Environment]::GetEnvironmentVariable("AI_CONTRACT_ONCALL_NAME")
|
||||
}
|
||||
if ([string]::IsNullOrWhiteSpace($ResponderName)) {
|
||||
$ResponderName = [Environment]::UserName
|
||||
}
|
||||
if ([string]::IsNullOrWhiteSpace($ResponderName)) {
|
||||
Add-Check -Key "responder" -Label "responder identity" -Status "fail" -Detail "ResponderName or AI_CONTRACT_ONCALL_NAME is required"
|
||||
} else {
|
||||
Add-Check -Key "responder" -Label "responder identity" -Status "pass" -Detail "Responder identity recorded"
|
||||
}
|
||||
|
||||
$externalWebhook = [Environment]::GetEnvironmentVariable($WebhookUrlEnvVar)
|
||||
if ([string]::IsNullOrWhiteSpace($externalWebhook)) {
|
||||
if ($RequireExternalWebhook) {
|
||||
Add-Check -Key "external_webhook" -Label "external webhook" -Status "fail" -Detail "$WebhookUrlEnvVar is not configured"
|
||||
} else {
|
||||
Add-Check -Key "external_webhook" -Label "external webhook" -Status "warn" -Required $false -Detail "$WebhookUrlEnvVar is not configured; external receipt remains blocked"
|
||||
}
|
||||
} else {
|
||||
Add-Check -Key "external_webhook" -Label "external webhook" -Status "pass" -Detail "$WebhookUrlEnvVar is configured"
|
||||
}
|
||||
|
||||
$status = "UNKNOWN"
|
||||
$severity = "unknown"
|
||||
$generatedAt = $null
|
||||
$alertArtifactDir = ""
|
||||
if ($sourceData) {
|
||||
if ($source.type -eq "payload") {
|
||||
if ($sourceData.status) { $status = [string]$sourceData.status }
|
||||
if ($sourceData.severity) { $severity = [string]$sourceData.severity }
|
||||
if ($sourceData.generated_at) { $generatedAt = [datetime]$sourceData.generated_at }
|
||||
if ($sourceData.artifact_dir) { $alertArtifactDir = [string]$sourceData.artifact_dir }
|
||||
} else {
|
||||
if ($sourceData.status) { $status = [string]$sourceData.status }
|
||||
if ($sourceData.alert_severity) { $severity = [string]$sourceData.alert_severity }
|
||||
if ($sourceData.generated_at) { $generatedAt = [datetime]$sourceData.generated_at }
|
||||
if ($sourceData.artifact_dir) { $alertArtifactDir = [string]$sourceData.artifact_dir }
|
||||
}
|
||||
}
|
||||
|
||||
$route = Get-ResponseRoute -Status $status -Severity $severity
|
||||
$ackAt = Get-Date
|
||||
$ackAgeMinutes = $null
|
||||
if ($EvaluateHistoricalSla -and $generatedAt) {
|
||||
$ackAgeMinutes = [math]::Round(($ackAt.ToUniversalTime() - $generatedAt.ToUniversalTime()).TotalMinutes, 2)
|
||||
if ($route.ack_sla_minutes -gt 0 -and $ackAgeMinutes -gt $route.ack_sla_minutes) {
|
||||
Add-Check -Key "ack_sla" -Label "ack SLA" -Status "warn" -Required $false -Detail "Historical ack age ${ackAgeMinutes}m exceeds SLA $($route.ack_sla_minutes)m"
|
||||
} else {
|
||||
Add-Check -Key "ack_sla" -Label "ack SLA" -Status "pass" -Detail "Historical ack age ${ackAgeMinutes}m is within SLA"
|
||||
}
|
||||
} else {
|
||||
Add-Check -Key "ack_sla" -Label "ack SLA" -Status "pass" -Detail "Drill acknowledgment recorded at run time"
|
||||
}
|
||||
|
||||
$record = [ordered]@{
|
||||
incident_id = "watchdog-$timestamp"
|
||||
action = $Action
|
||||
responder = $ResponderName
|
||||
response_note = $ResponseNote
|
||||
acknowledged_at = $ackAt.ToUniversalTime().ToString("o")
|
||||
source = $source
|
||||
alert = [ordered]@{
|
||||
status = $status
|
||||
severity = $severity
|
||||
generated_at = if ($generatedAt) { $generatedAt.ToUniversalTime().ToString("o") } else { $null }
|
||||
artifact_dir = $alertArtifactDir
|
||||
}
|
||||
route = $route
|
||||
external_receipt = [ordered]@{
|
||||
webhook_env_var = $WebhookUrlEnvVar
|
||||
configured = -not [string]::IsNullOrWhiteSpace($externalWebhook)
|
||||
verified_to_person = $false
|
||||
}
|
||||
}
|
||||
Save-Json -Path $responseRecordJson -Value $record
|
||||
|
||||
$recordMd = New-Object System.Collections.Generic.List[string]
|
||||
$recordMd.Add("# Watchdog Alert Response Record") | Out-Null
|
||||
$recordMd.Add("") | Out-Null
|
||||
$recordMd.Add("- Incident: $($record.incident_id)") | Out-Null
|
||||
$recordMd.Add("- Action: $Action") | Out-Null
|
||||
$recordMd.Add("- Responder: $ResponderName") | Out-Null
|
||||
$recordMd.Add("- Alert status: $status") | Out-Null
|
||||
$recordMd.Add("- Alert severity: $severity") | Out-Null
|
||||
$recordMd.Add("- Alert artifact: $alertArtifactDir") | Out-Null
|
||||
$recordMd.Add("- Response note: $ResponseNote") | Out-Null
|
||||
$recordMd.Add("") | Out-Null
|
||||
$recordMd.Add("First actions:") | Out-Null
|
||||
foreach ($item in $route.first_actions) {
|
||||
$recordMd.Add("- $item") | Out-Null
|
||||
}
|
||||
$recordMd | Set-Content -Encoding UTF8 -Path $responseRecordMd
|
||||
|
||||
$failedRequired = @($checks | Where-Object { $_.required -and $_.status -eq "fail" })
|
||||
$warningsList = @($checks | Where-Object { $_.status -eq "warn" })
|
||||
$overallStatus = if ($failedRequired.Count -gt 0) { "FAIL" } elseif ($warningsList.Count -gt 0) { "WARN" } else { "PASS" }
|
||||
$alertSeverity = if ($failedRequired.Count -gt 0) { "critical" } elseif ($warningsList.Count -gt 0) { "warning" } else { "ok" }
|
||||
|
||||
$summary = [ordered]@{
|
||||
name = "phase7.7-alert-response-drill"
|
||||
status = $overallStatus
|
||||
alert_severity = $alertSeverity
|
||||
generated_at = (Get-Date).ToUniversalTime().ToString("o")
|
||||
artifact_dir = $artifactDir
|
||||
record = $record
|
||||
checks = $checks
|
||||
totals = [ordered]@{
|
||||
total = $checks.Count
|
||||
failed_required = $failedRequired.Count
|
||||
warnings = $warningsList.Count
|
||||
}
|
||||
}
|
||||
Save-Json -Path $summaryJson -Value $summary
|
||||
|
||||
$markdown = New-Object System.Collections.Generic.List[string]
|
||||
$markdown.Add("# Phase 7.7 Alert Response Drill") | Out-Null
|
||||
$markdown.Add("") | Out-Null
|
||||
$markdown.Add("- Status: $overallStatus") | Out-Null
|
||||
$markdown.Add("- Alert severity: $alertSeverity") | Out-Null
|
||||
$markdown.Add("- Generated at: $($summary.generated_at)") | Out-Null
|
||||
$markdown.Add("- Artifact dir: $artifactDir") | Out-Null
|
||||
$markdown.Add("- Incident: $($record.incident_id)") | Out-Null
|
||||
$markdown.Add("- Responder: $ResponderName") | Out-Null
|
||||
$markdown.Add("- Alert status: $status") | Out-Null
|
||||
$markdown.Add("- Alert route: $($route.route)") | Out-Null
|
||||
$markdown.Add("") | Out-Null
|
||||
$markdown.Add("| Check | Required | Status | Detail |") | Out-Null
|
||||
$markdown.Add("|---|---:|---|---|") | Out-Null
|
||||
foreach ($check in $checks) {
|
||||
$markdown.Add("| $($check.label) | $($check.required) | $($check.status.ToUpperInvariant()) | $($check.detail) |") | Out-Null
|
||||
}
|
||||
$markdown.Add("") | Out-Null
|
||||
$markdown.Add("Local artifacts:") | Out-Null
|
||||
$markdown.Add("") | Out-Null
|
||||
$markdown.Add('- `alert-response-summary.json`') | Out-Null
|
||||
$markdown.Add('- `alert-response-summary.md`') | Out-Null
|
||||
$markdown.Add('- `alert-response-record.json`') | Out-Null
|
||||
$markdown.Add('- `alert-response-record.md`') | Out-Null
|
||||
$markdown | Set-Content -Encoding UTF8 -Path $summaryMd
|
||||
|
||||
Write-Host "PHASE7_7_ALERT_RESPONSE_DRILL_$overallStatus $artifactDir"
|
||||
if ($overallStatus -eq "FAIL" -or ($FailOnWarn -and $overallStatus -eq "WARN")) {
|
||||
throw "Phase 7.7 alert response drill $overallStatus. See $summaryMd"
|
||||
}
|
||||
Reference in New Issue
Block a user