feat: add privileged identity outbox observability
Build and Test Quality Gate / Contract Web typecheck and build (pull_request) Successful in 13m17s
Build and Test Quality Gate / Python tests and Contract API migration (pull_request) Failing after 29m27s
Security Quality Gate / npm audit, Bandit, pip-audit, detect-secrets (pull_request) Failing after 23m1s
Build and Test Quality Gate / Contract Web typecheck and build (pull_request) Successful in 13m17s
Build and Test Quality Gate / Python tests and Contract API migration (pull_request) Failing after 29m27s
Security Quality Gate / npm audit, Bandit, pip-audit, detect-secrets (pull_request) Failing after 23m1s
This commit is contained in:
@@ -683,3 +683,103 @@ P1 发布门槛：五类操作均可在受控运营台完成端到端演练；
|
||||
- Alembic 代码 head 为 `20260802_0039`。真实 PostgreSQL 专项使用临时 schema 验证不可变触发器，测试结束后清理，不把测试迁移或事件表遗留在公共 schema。
|
||||
|
||||
本检查点完成后仍不得把审批状态机扩展为直接身份写入。下一阶段必须以独立设计和验收任务建设统一执行器、固定锁顺序、执行前版本复核、outbox、失败补偿、支持访问在线鉴权/到期/紧急撤销，以及独立回滚预览和双人审批。
|
||||
|
||||
## 21. Transactional outbox 最小切片检查点（2026-08-02）
|
||||
|
||||
本阶段新增迁移 `20260802_0040` 和 `privileged_identity_outbox_messages`，但不改变身份执行边界。提交、批准、拒绝和过期物化在追加状态事件的同一数据库事务内生成一条 outbox 消息；`(event_id, effect_type)` 与 `idempotency_key` 均有唯一约束，因此操作或审批的幂等重放不会重复排队。
|
||||
|
||||
- 迁移不为 `0038/0039` 已存在的历史事件回填 outbox，避免把历史状态重新解释为待投递副作用；投递状态只覆盖 0040 上线后新产生的事件。
|
||||
- outbox payload 只包含事件/操作 ID、序号、类型、状态、计数、目标指纹、安全 `trace_id`、快照哈希和发生时间，不包含完整快照、操作者、邮箱、姓名、工单引用或正文、凭据、令牌、密码、secret 或原始目标引用；`payload_hash` 可由 payload 重算。
|
||||
- worker 基座采用 at-least-once 语义：PostgreSQL 使用 `FOR UPDATE SKIP LOCKED` 认领；消息认领后持有有限租约，失败按指数退避重试，达到最大次数进入 `dead_letter`。SQLite 领取使用 `BEGIN IMMEDIATE` 防止意外重复认领，但部署仍只支持单进程、单 worker，不作为横向扩容方案。
|
||||
- 平台管理员仅可调用 `GET /api/v1/identity/privileged-operations/{operation_id}/delivery-status` 查看消息 ID、事件、状态、尝试次数、稳定错误码和时间；响应不返回 `payload`、`lease_owner` 或错误详情。
|
||||
- 当前 transport 默认为禁用且不领取消息；没有 SMTP、工单或其他外部副作用，没有人工 `POST retry`，也没有 execute、rollback、merge、split、recover、grant、revoke 或真实 support access 写入口。
|
||||
- 发布采用先迁移后发代码；代码回滚时保留 outbox 表。执行 `0040` 数据库 downgrade 前必须停止 worker，并确认不存在 queued、processing、retry_scheduled 或 dead-letter 消息，避免丢失投递意图或事故证据。
|
||||
- SQLite 迁移与治理专项为 `18/18`，包含提交、批准、拒绝、过期故障注入、delivery-status GET 无写副作用和底层并发认领互斥；页面专项 Node `2/2`、Web 全量 Node `34/34`、TypeScript、Python compileall 和 101 路由生产构建已通过。PostgreSQL 专项已在 `192.168.3.90:5432/yuqei_contract` 的隔离临时 schema 中 `1/1` 通过，覆盖 0040 head、事件/outbox 一一对应、并发领取互斥、租约恢复、稳定幂等键、重试和 dead-letter；测试 schema 已精确清理。
|
||||
|
||||
下一阶段仍须单独评审 transport 选择、消息消费者幂等契约、告警与死信处置 SOP；启用真实 transport 前，还必须评审客户端可控 operation ID 与无密钥目标指纹的外发关联风险，必要时改用服务端 opaque ID/HMAC 或移除非必要指纹。上述能力不得被解释为身份执行器已经获准开放。
|
||||
|
||||
## 22. 全局 outbox 只读可观测与死信排查交付检查点（2026-08-02）
|
||||
|
||||
### 22.1 目标与边界
|
||||
|
||||
本阶段只补齐平台管理员对 0040 outbox 的全局健康判断和死信排查能力，复用现有 `privileged_identity_outbox_messages`，不新增 `0041` 迁移。所有入口均为只读 `GET`，不得认领、续租、发送、重试、重排、确认或删除消息，也不得物化治理操作过期状态。
|
||||
|
||||
- 活跃消息统一指 `queued`、`processing`、`retry_scheduled`。
|
||||
- 默认超龄阈值为 900 秒；只读查询可通过受限的 `stale_after_seconds` 调整阈值，但不得改变数据库配置或消息状态。
|
||||
- 临近最大尝试指活跃消息的 `attempt_count >= max_attempts - 1`；过期租约指 `processing` 消息存在租约且 `lease_expires_at` 早于查询时刻。
|
||||
- 健康分级按最高风险优先：存在任意 `dead_letter` 或过期 processing 租约为 `critical`；否则存在超龄活跃消息或临近最大尝试消息为 `degraded`；其余为 `healthy`。空队列必须为 `healthy`。
|
||||
- 最老活跃消息年龄在没有活跃消息时返回 `null`，不能伪造为 `0`；所有阈值计算复用同一个 `generated_at` 查询时刻。真实 PostgreSQL 专项已在隔离临时 schema 中验证并发读一致性，以及 overview/dead-letter 两个只读查询前后 operation、approval、event、outbox 全字段快照完全一致。
|
||||
|
||||
### 22.2 API 与脱敏契约
|
||||
|
||||
使用与单操作详情分离的命名空间，避免和 `{operation_id}` 路由冲突：
|
||||
|
||||
- `GET /api/v1/identity/privileged-outbox/overview`
|
||||
- `GET /api/v1/identity/privileged-outbox/dead-letters`
|
||||
|
||||
两个接口仅允许当前仍有效的平台管理员访问；个人用户、企业普通成员、企业管理员和已撤销平台权限的旧会话均返回 `403`，未登录返回 `401`。接口实现不得调用 `_materialize_expired_operations` 或任何 worker claim/transition 方法。
|
||||
|
||||
`overview` 至少返回查询时刻、健康级别、各状态计数、活跃总数、最老活跃消息年龄、超龄活跃数、过期 processing 租约数、临近最大尝试数，以及按稳定错误码汇总的 dead-letter 数量。空白或缺失错误码统一归入固定的 `unknown`，不把原始错误详情转换为展示文本。
|
||||
|
||||
`dead-letters` 提供稳定分页、稳定错误码过滤和确定性排序。默认按 `updated_at DESC, message_id DESC` 排序，分页参数有明确上限；operation type 和 effect type 作为只读排查上下文返回，本切片不提供对应组合筛选。单项只允许返回：
|
||||
|
||||
- message、event、operation ID；
|
||||
- event type、operation type、effect type；
|
||||
- 安全 `trace_id`；
|
||||
- attempt/max attempts；
|
||||
- 稳定错误码；
|
||||
- created、updated、last attempt 时间。
|
||||
|
||||
两个响应均不得返回 payload、payload hash、idempotency key、lease owner、lease token、原始错误详情、目标指纹、快照或快照哈希、actor、邮箱、姓名、账号/成员身份信息、工单正文、凭据或 secret。列表和聚合也不得通过筛选项、错误文案或导出功能间接泄露这些字段。
|
||||
|
||||
### 22.3 只读一致性与验收标准
|
||||
|
||||
- [x] 平台管理员访问 overview 和 dead-letters 返回 `200`；普通用户和企业管理员返回 `403`，未登录返回 `401`。
|
||||
- [x] 空队列返回 `healthy`，所有状态计数为 0，最老活跃消息年龄为 `null`，dead-letter 列表为空页。
|
||||
- [x] queued、processing、retry_scheduled、过期租约、超龄、临近最大尝试和 dead-letter 的计数与健康优先级符合 22.1，阈值等于边界值时有固定且已测试的包含/排除语义。
|
||||
- [x] dead-letter 按稳定错误码聚合准确；空错误码只进入 `unknown`，响应不包含原始错误详情。
|
||||
- [x] dead-letter 分页无重复、无遗漏，稳定错误码过滤准确，时间相同记录仍按 message ID 确定性排序，非法页码、页大小和阈值返回稳定 `422`。
|
||||
- [x] 对同一数据库连续调用两个 GET 前后，operation、approval、event 和 outbox 的行数、状态、attempt、lease、时间戳、payload 与 hash 完全不变；不得因查询物化过期操作或追加事件/outbox。
|
||||
- [x] 响应字段与序列化后的 JSON 均通过敏感字段泄露扫描，不出现 payload、幂等键、租约所有者、原始错误、目标指纹或身份信息。
|
||||
- [x] transport disabled 时全局只读接口仍可用，worker 发送/认领数量保持 0，调用接口不会触发 claim。
|
||||
- [x] OpenAPI 只新增上述两个 GET，不出现 `POST retry/requeue/ack`，也不出现 execute、rollback、merge、split、recover、grant、revoke 或 support access 身份写路径。
|
||||
- [x] SQLite 定向测试、Web 定向测试、TypeScript、Python 编译、生产构建和 `git diff --check` 通过。
|
||||
- [x] PostgreSQL 0040 并发认领、租约恢复和本阶段只读一致性查询已在 `192.168.3.90:5432/yuqei_contract` 的隔离临时 schema 中 `1/1` 通过；测试完成后 schema 已精确清理，未以 SQLite 结果替代。
|
||||
|
||||
本地交付证据：治理与迁移专项 `20/20`、反滥用专项 `9/9`、Web 全量 Node `36/36`、页面专项 `4/4`、真实 PostgreSQL 0040/并发/租约/死信/只读一致性专项 `1/1`、TypeScript、Python compileall、单一 Alembic head `20260802_0040`、101 路由生产构建和 `git diff --check` 均通过。运行中 OpenAPI 对两个新路径各注册 1 个 GET，outbox 写方法数量为 0；已登录平台管理员桌面/390px 移动视图加载成功且控制台 error 为 0。Contract API 全量测试未在本检查点复跑，不能用上述定向结果替代。
|
||||
|
||||
### 22.4 死信排查 SOP
|
||||
|
||||
1. 值班人员只读打开 overview，记录查询时刻、健康级别、dead-letter/过期租约/超龄/临近上限计数和使用的阈值；`critical` 立即建立 P0/P1 事故记录并通知身份治理与平台值班负责人，`degraded` 按 P1 运营异常跟踪。
|
||||
2. 在 dead-letters 中按稳定错误码、operation type 和 effect type 缩小范围，记录 message/event/operation ID、安全 `trace_id`、attempt/max attempts 与时间，不复制 payload、目标引用、actor 或原始错误到工单。
|
||||
3. 使用安全 `trace_id` 和三个不透明 ID 在受控日志/审计系统交叉核对部署版本、数据库连接、消费者版本和外部依赖状态；需要查看受限原始日志时必须走既有最小权限与审计流程，不能扩展本页面响应。
|
||||
4. 判断是否为代码缺陷、配置/依赖故障、租约异常或消费者幂等冲突。禁止直接 `UPDATE/DELETE` outbox、修改 attempt/lease、手工重排、人工调用 transport 或绕过审批执行身份写入。
|
||||
5. 修复方案通过独立变更评审和回归后发布。当前没有人工 retry/ack 能力，历史 dead-letter 保留为事故证据；若未来需要恢复投递，必须另立设计，定义授权、幂等、审计和回滚并重新通过安全评审。
|
||||
6. 发布后用 overview 观察新消息是否继续进入异常状态，并在事故记录中附只读查询证据。关闭事故不删除或改写 dead-letter；涉及 PostgreSQL 多 worker 的结论必须引用隔离 schema 并发验收记录。
|
||||
|
||||
### 22.5 本阶段明确非目标与发布门禁
|
||||
|
||||
- 不新增数据库迁移，不改变 0040 状态机、约束、payload 或历史不回填策略。
|
||||
- 不增加 `POST retry`、requeue、ack、cancel、delete、lease takeover 或任何人工消息状态写接口。
|
||||
- 不启用 worker transport，不连接 SMTP、工单、Webhook 或其他外部消费者，不把可观测页面作为 transport 上线批准。
|
||||
- 不实现消费者业务副作用、身份执行器、支持访问授权、失败补偿或回滚；审批完成仍不代表身份数据已变更。
|
||||
- 不开放 execute、rollback、merge、split、recover、grant、revoke，不直接写账号、成员、凭据、平台权限或工作空间归属。
|
||||
- 真实隔离 PostgreSQL 0040 并发验证现已通过，但该结果不等于外部副作用获准上线。完成 transport 选择、消费者幂等契约和外发关联风险评审前，多 worker 与真实 transport 继续阻断。
|
||||
|
||||
## 23. Outbox 数据库完整性加固检查点（2026-08-03）
|
||||
|
||||
本检查点是 22 节只读可观测交付之后的独立数据库加固阶段。22.5 中“不新增 `0041`”仅描述当时不改变 0040 模型的阶段边界；本阶段根据独立质量复核发现的跨操作事件归属和非法租约组合风险，新增迁移 `20260803_0041`，不回改已经验收的 0040 历史迁移。
|
||||
|
||||
- 升级前先只读统计 outbox 与 event 的 operation 归属错配，以及 status/lease 非法组合。任一计数非零时拒绝迁移，不自动修复、删除或重排数据；跨操作错配必须作为审计完整性事件调查。
|
||||
- 事件表增加唯一索引 `(operation_id, event_id)`；outbox 增加 `(operation_id, event_id)` 复合外键并保持 `ON DELETE NO ACTION`。PostgreSQL 由外键直接拒绝把操作 A 的消息关联到操作 B 的事件；SQLite 默认可能关闭外键执行，因此迁移与自动建库路径同时安装 outbox INSERT/UPDATE 拒绝触发器，不能只依赖外键声明。
|
||||
- 新增租约 CHECK：`processing` 必须同时具有 `lease_owner` 和 `lease_expires_at`；任何非 `processing` 状态必须同时清空两项。该约束避免 processing 永久滞留且无法被重领/告警，也避免 queued、retry、delivered 或 dead-letter 残留租约。
|
||||
- SQLite batch-recreate outbox 表并安装两条关联拒绝触发器，事件表只增加唯一索引；0039 的事件 UPDATE/DELETE 拒绝触发器保留。PostgreSQL 增加索引和约束时可能取得表锁，发布时必须短暂停止 Contract API 治理写流量；当前 transport 已关闭，不存在 worker 写流量。
|
||||
- downgrade 只移除复合外键、租约 CHECK 和事件唯一索引，不删除或改写 operation、approval、event、outbox 数据。若需要回滚，仍应先停止治理写流量并保留审计证据。
|
||||
|
||||
验收证据：SQLite 0041 迁移专项 `2/2` 通过，覆盖两类升级前拒绝、复合外键、六种非法租约组合、四种合法租约状态及 downgrade/upgrade；治理与 worker 专项 `19/19` 通过。真实 PostgreSQL 隔离 schema 专项 `1/1` 通过，覆盖 0041 head、约束元数据、跨操作关联拒绝、非法租约拒绝、并发 `SKIP LOCKED`、租约恢复、重试/dead-letter、只读可观测无副作用和 schema 精确清理；组合专项合计 `22/22`。
|
||||
|
||||
Contract API 单进程全量回归在 10 分钟命令上限处被终止且未返回失败堆栈，因此不把该次运行写成通过。随后按四个隔离进程覆盖全部 24 个测试模块，收集并执行 `234/234` 项全部通过；各组分别为合同工作流 `91/91`、身份与授权 `83/83`、0041/特权身份/PostgreSQL `22/22`、基础设施与迁移 `38/38`。Python compileall、单一 Alembic head `20260803_0041` 和 `git diff --check` 通过。
|
||||
|
||||
本地运行 SQLite 库升级前为 0040，outbox 行数、跨操作事件错配和非法租约计数均为 `0`；停止 Contract API 后已生成 `contract-api-20260802-before-0041-20260803-012502.sqlite3` 备份，再升级到 0041。补充 SQLite 触发器后执行 0041 downgrade/upgrade 循环，确认复合索引、复合外键、租约 CHECK 和两条关联拒绝触发器均存在；Contract API 重启后 `8110` 健康检查返回 `200`，Web `3021` 与 AI Platform `8111` 未重启。
|
||||
|
||||
本阶段没有新增或改变任何 API、DTO、Web 页面或 worker transport，不增加人工 retry/requeue/ack，不启用 SMTP/工单/Webhook，不开放 execute、rollback、merge、split、recover、grant、revoke、support access 或其他身份执行入口。审批完成仍只表示治理审批状态，不代表身份数据已变更。
|
||||
|
||||
@@ -518,7 +518,7 @@ legal_name: string | null
|
||||
|
||||
### 9.28 第十阶段验证证据与后续优先项
|
||||
|
||||
- SQLite 治理与迁移专项 `9/9` 通过；Contract API 全量 `220 passed, 1 skipped`，跳过项为需要显式连接参数的 PostgreSQL 专项。该 PostgreSQL 专项随后在 `192.168.3.90:15432` 的唯一临时 schema 中单独执行并 `1/1` 通过。
|
||||
- SQLite 治理与迁移专项 `9/9` 通过；Contract API 全量 `220 passed, 1 skipped`，跳过项为需要显式连接参数的 PostgreSQL 专项。该 PostgreSQL 专项随后在 `192.168.3.90:5432/yuqei_contract` 的唯一临时 schema 中单独执行并 `1/1` 通过。
|
||||
- 真实 PostgreSQL 证据覆盖：并发相同操作只创建 1 条操作和 1 条提交事件；同一审批人并发只创建 1 条审批和 1 条决定事件；两个不同审批人并发形成两条有序事件并最终进入 `approved`；拒绝保持终态；过期只物化一次；事件序号、`trace_id`、快照哈希和脱敏规则均通过；最后有效平台管理员撤销预览继续被阻断。
|
||||
- PostgreSQL 测试按阶段运行 Alembic 到单一 head `20260802_0038`，仅在 `pg_trgm` 索引迁移期间临时把 `public` 加入 `search_path`，其余迁移和运行时连接只使用测试 schema。夹具在 `finally` 中精确删除测试 schema。诊断试跑曾只推进公共 Alembic 标记而未创建事件表；已恢复公共标记为原基线 `20260802_0037`，确认公共事件表不存在、残留治理/诊断 schema 数量为 `0`。
|
||||
- Web 治理定向回归 `28/28`、TypeScript 和 Python 编译通过；Next.js 生产构建成功生成 `101` 个页面/路由；`git diff --check` 无错误，仅有既有 LF/CRLF 提示。Alembic 当前代码 head 为 `20260802_0038`。
|
||||
@@ -531,3 +531,56 @@ legal_name: string | null
|
||||
- 独立质量复核后新增迁移 `20260802_0039`：SQLite 使用两个拒绝触发器、PostgreSQL 使用单一 `BEFORE UPDATE OR DELETE` 触发器，数据库层直接修改或删除治理事件均失败；本地自动建库路径同步安装等价保护。列表与详情也统一映射过期物化等治理异常，避免极端并发冲突退化为 `500`。
|
||||
- 最终本地复核新增执行 Web 全部 `*.test.mjs`，结果 `34/34`；治理迁移、不可变触发器、错误映射与操作专项在隔离内存数据库下 `10/10`；TypeScript、Python 编译、101 路由 Next.js 生产构建和 `git diff --check` 通过。Web `3021`、Contract API `8110`、AI Platform `8111` 均返回 `200`，Alembic 代码为单一 head `20260802_0039`。
|
||||
- 本阶段交付结论不变：追加式事件、追踪、脱敏快照和真实 PostgreSQL 并发门槛已完成；身份执行器仍为关闭状态。下一阶段不得直接开放破坏性写入口，必须先单独评审统一执行器、固定锁顺序、版本复核、outbox、失败补偿、支持访问在线鉴权和独立回滚治理。
|
||||
|
||||
### 9.30 第十一阶段 transactional outbox 最小切片（2026-08-02）
|
||||
|
||||
- 新增 `20260802_0040` 加法迁移、ORM 和 `privileged_identity_outbox_messages`。操作与事件外键均为 `NO ACTION`；状态、尝试次数、唯一事件/effect 和唯一幂等键由数据库约束，投递扫描与操作详情均有索引。
|
||||
- 0040 不回填 0038/0039 的历史事件，避免旧治理状态在部署后被重新投递；投递状态从升级后新事件开始记录。
|
||||
- 提交、批准、拒绝和过期事件与脱敏 outbox 消息在同一事务写入。相同操作或审批幂等重放不新增事件或消息；消息 payload 不包含原始身份目标、操作者信息或凭据类字段。
|
||||
- worker repository 已具备 PostgreSQL `FOR UPDATE SKIP LOCKED`、租约、过期重领、指数退避、最大重试和 `dead_letter` 基座；SQLite 使用 `BEGIN IMMEDIATE` 防重复领取且部署明确限制单进程单 worker。当前未接任何外部 transport。
|
||||
- 平台管理员详情新增只读 `delivery-status`，前端只展示投递状态和尝试次数；API 不返回 payload、lease owner 或错误详情，且未增加发送、重试或执行控件。
|
||||
- 回归门槛：SQLite 治理与迁移专项 `18/18`（包含底层并发认领互斥）、页面专项 Node `2/2`、Web 全量 Node `34/34`、TypeScript、Python compileall 和 101 路由生产构建已通过。PostgreSQL 专项已在 `192.168.3.90:5432/yuqei_contract` 的隔离临时 schema 中 `1/1` 通过，校验 0040 head、outbox/事件数量一致、并发认领不重复、租约恢复、稳定幂等键、重试和 dead-letter；测试 schema 已精确清理。
|
||||
- 身份执行器继续关闭；execute、rollback、merge、split、recover、grant、revoke、真实 support access 写入及人工 outbox retry 仍属禁止范围。
|
||||
|
||||
### 9.31 第十二阶段：全局 outbox 只读可观测与死信排查（已交付，2026-08-02）
|
||||
|
||||
本阶段优先级为 **P1 运营可观测**，仅复用 0040 数据模型，不新增迁移。目标是让有效平台管理员在不触发任何数据库写入、消息认领或外部副作用的前提下判断 outbox 全局健康并定位 dead-letter；它不是 transport、人工重试或身份执行器的发布阶段。
|
||||
|
||||
可执行 Backlog：
|
||||
|
||||
1. **P1-OBS-1 后端只读聚合（完成）**：已增加状态计数、活跃总数、最老活跃消息年龄、超龄活跃数、过期 processing 租约数、临近最大尝试数和 dead-letter 稳定错误码聚合。活跃状态固定为 `queued/processing/retry_scheduled`，默认超龄阈值 900 秒。
|
||||
2. **P1-OBS-2 独立只读 API（完成）**：已增加 `GET /api/v1/identity/privileged-outbox/overview` 和 `GET /api/v1/identity/privileged-outbox/dead-letters`；只允许实时有效的平台管理员，禁止调用过期物化、claim、retry 或状态迁移方法。
|
||||
3. **P1-OBS-3 死信分页排查（完成）**：已按 `updated_at DESC, message_id DESC` 稳定排序，支持稳定错误码过滤和有上限的分页；operation type、effect type 作为只读上下文返回，本切片未增加对应组合筛选。响应只返回不透明关联 ID、安全 `trace_id`、类型、尝试次数、稳定错误码和时间字段。
|
||||
4. **P1-OBS-4 前端只读运营区（完成）**：特权身份治理页已增加全局投递健康摘要和 dead-letter 排查区，覆盖 loading、healthy/degraded/critical、空队列、空筛选结果、分页和后端失败状态；不存在 retry、ack、requeue、send、execute 或其他消息/身份动作按钮。
|
||||
5. **P1-OBS-5 回归与泄露门禁（完成）**：已覆盖平台管理员 `200`、个人/企业管理员 `403`、空队列、阈值边界、健康优先级、聚合、稳定分页/过滤/排序、GET 前后数据库完全不变、禁用 transport 不 claim、OpenAPI 破坏性路径扫描、S3 反滥用策略和响应敏感字段扫描。
|
||||
6. **P1-OBS-6 PostgreSQL 证据补齐（完成）**：已在 `192.168.3.90:5432/yuqei_contract` 的隔离临时 schema 中复跑并 `1/1` 通过，覆盖 0040 迁移、并发认领、租约恢复、重试/dead-letter、稳定错误码聚合/过滤，以及 overview/dead-letter 查询前后 operation、approval、event、outbox 全字段快照完全一致；测试后已精确清理 schema。
|
||||
|
||||
本地验收结果：治理与迁移专项 `20/20`、反滥用专项 `9/9`、Web 全量 Node `36/36`、页面专项 `4/4`、真实 PostgreSQL 0040/并发/租约/死信/只读一致性专项 `1/1`、TypeScript、Python compileall、单一 Alembic head `20260802_0040`、101 路由生产构建及 `git diff --check` 通过；运行时两个新 GET 各注册 1 条，outbox 写方法数量为 0，桌面与 390px 移动视图加载成功且浏览器控制台 error 为 0。Contract API 全量测试未在本检查点复跑，仍作为独立发布证据缺口保留。
|
||||
|
||||
验收标准：
|
||||
|
||||
- 空队列为 `healthy`；存在 dead letter 或过期 processing 租约为 `critical`；否则存在超龄活跃消息或临近最大尝试消息为 `degraded`；其余为 `healthy`。
|
||||
- 两个 GET 前后 operation、approval、event、outbox 的行数、状态、attempt、lease、时间戳、payload 和 hash 完全不变；实现不得调用 `_materialize_expired_operations`。
|
||||
- 响应不得包含 payload、payload hash、idempotency key、lease owner/token、原始错误详情、目标指纹、快照/哈希、actor、邮箱、姓名、账号/成员身份、工单正文、凭据或 secret；空错误码聚合为固定 `unknown`。
|
||||
- dead-letter 分页无重复或遗漏，组合过滤准确，同时间记录排序确定；非法分页和阈值参数返回稳定 `422`。
|
||||
- transport disabled 时发送和认领数量继续为 0；OpenAPI 只新增两个 GET，破坏性身份写路径和人工 outbox 写路径数量均为 0。
|
||||
- 后端和 Web 定向测试、TypeScript、Python 编译、生产构建及 `git diff --check` 通过；PostgreSQL 专项必须单列真实结果，本阶段真实结果为 `1/1` 通过且测试 schema 已清理，未以 SQLite 结果代替。
|
||||
|
||||
运营 SOP：先在 overview 记录查询时刻、阈值和风险计数，再在 dead-letters 按稳定错误码及类型过滤，使用安全 `trace_id` 和不透明 ID 在受控日志中关联部署/消费者/依赖证据。不得把 payload、原始目标、actor 或错误详情复制到普通工单，不得直接修改 outbox 或人工发送。`critical` 立即升级身份治理与平台值班负责人；修复必须走独立代码/配置变更评审，历史 dead-letter 继续保留为证据，当前阶段不提供 retry/ack。
|
||||
|
||||
明确非目标：不新增 `0041`，不回填历史事件，不启用 transport，不接 SMTP/工单/Webhook，不新增 POST retry/requeue/ack，不开放身份执行器、支持访问或 execute/rollback/merge/split/recover/grant/revoke。真实 PostgreSQL 0040 并发验证虽已完成，但在 transport 选择、消费者幂等契约和外发关联风险评审完成前，仍不得推进多 worker 或真实副作用投递。
|
||||
|
||||
### 9.32 第十三阶段：outbox 数据完整性加固（已交付，2026-08-03）
|
||||
|
||||
9.31 的“不新增 `0041`”是只读可观测阶段的范围约束。本阶段根据独立质量复核结果，以新的加法迁移 `20260803_0041` 单独处理两个数据库完整性缺口，不修改 0040 历史迁移，也不扩大产品能力。
|
||||
|
||||
1. **P1-INT-1 事件归属复合约束（完成）**：事件表增加唯一 `(operation_id, event_id)` 索引；outbox 增加同列复合外键并使用 `ON DELETE NO ACTION`。SQLite 默认可能关闭外键执行，迁移与自动建库路径另安装 INSERT/UPDATE 拒绝触发器；PostgreSQL 由复合外键执行。两种数据库均拒绝跨 operation/event 拼接，避免详情、死信上下文和未来消费者看到错误审计归属。
|
||||
2. **P1-INT-2 租约状态约束（完成）**：`processing` 必须同时存在 lease owner/expiry，其他状态必须同时清空两项。暂不扩大到 delivered timestamp、last attempt 或 error code 状态机，避免在未完成设计前增加额外语义。
|
||||
3. **P1-INT-3 升级预检与回滚（完成）**：0041 升级最先只读统计跨操作错配和非法租约行；任一计数非零立即拒绝，不自动修复、删除或重排。downgrade 只移除本阶段约束和索引，不改写业务数据。
|
||||
4. **P1-INT-4 SQLite/PostgreSQL 一致性（完成）**：SQLite batch-recreate outbox 并安装关联拒绝触发器，0039 事件不可变触发器继续有效；PostgreSQL 使用复合外键与 CHECK。发布时须短暂停止治理写流量，避免索引/约束验证锁与新治理提交竞争。
|
||||
5. **P1-INT-5 自动化证据（完成）**：SQLite 迁移 `2/2`、治理/worker `19/19`、真实 PostgreSQL `1/1`，组合专项 `22/22`。真实 PostgreSQL 覆盖 0041 head、约束元数据、非法关联/租约拒绝、并发 claim、租约恢复、重试/dead-letter、只读快照不变和隔离 schema 清理。
|
||||
6. **P1-INT-6 全模块回归（完成）**：Contract API 单进程运行在 10 分钟命令上限处终止且无失败堆栈，不据此宣称通过；随后四个隔离进程覆盖全部 24 个测试模块，收集并执行 `234/234` 项全部通过。分组结果为合同工作流 `91/91`、身份与授权 `83/83`、0041/特权身份/PostgreSQL `22/22`、基础设施与迁移 `38/38`。
|
||||
|
||||
当前单一 Alembic head 为 `20260803_0041`；Python compileall 和 `git diff --check` 通过。本阶段不新增 API、DTO、Web 或 transport 逻辑，身份执行器、真实 support access、人工 outbox retry 及 execute/rollback/merge/split/recover/grant/revoke 继续关闭。下一阶段仍须先完成 transport 选择、消费者幂等契约与外发关联风险评审，不得把数据库完整性加固解释为真实副作用或身份写入获准上线。
|
||||
|
||||
本地运行库已在停服备份后由 0040 升级到 0041；升级前 outbox、跨操作错配和非法租约计数均为 `0`，备份文件为 `.runtime/db-backups/contract-api-20260802-before-0041-20260803-012502.sqlite3`。补充 SQLite 触发器后已完成 0041 downgrade/upgrade 循环，复合索引、复合外键、租约 CHECK 和两条关联拒绝触发器复核通过；Contract API 重启并在 `8110` 返回 `200`，Web `3021` 与 AI Platform `8111` 保持运行，现有 Web 会话未因本次迁移重启。
|
||||
|
||||
+45
@@ -3,6 +3,10 @@ import { readFile } from "node:fs/promises";
|
||||
import test from "node:test";
|
||||
|
||||
const pageSource = await readFile(new URL("./page.tsx", import.meta.url), "utf8");
|
||||
const clientSource = await readFile(
|
||||
new URL("../../../../services/contracts-client.ts", import.meta.url),
|
||||
"utf8",
|
||||
);
|
||||
|
||||
test("the governance page exposes preview, records, and approval without execution", () => {
|
||||
assert.match(pageSource, /from "@\/services\/contracts-client"/);
|
||||
@@ -11,6 +15,9 @@ test("the governance page exposes preview, records, and approval without executi
|
||||
assert.match(pageSource, /createPrivilegedIdentityOperation/);
|
||||
assert.match(pageSource, /decidePrivilegedIdentityOperation/);
|
||||
assert.match(pageSource, /getPrivilegedIdentityOperationEvents/);
|
||||
assert.match(pageSource, /getPrivilegedIdentityOperationDeliveryStatus/);
|
||||
assert.match(pageSource, /事件投递/);
|
||||
assert.match(pageSource, /投递状态/);
|
||||
assert.match(pageSource, /状态事件时间线/);
|
||||
assert.match(pageSource, /trace_id/);
|
||||
assert.match(pageSource, /snapshot_after_hash/);
|
||||
@@ -18,6 +25,44 @@ test("the governance page exposes preview, records, and approval without executi
|
||||
assert.match(pageSource, /身份数据执行入口保持关闭/);
|
||||
assert.doesNotMatch(pageSource, /\/execute/);
|
||||
assert.doesNotMatch(pageSource, /\/merge|\/split|\/recover|\/grant|\/revoke/);
|
||||
assert.doesNotMatch(pageSource, /retryPrivileged|\/retry/);
|
||||
});
|
||||
|
||||
test("the governance page exposes read-only global outbox observability", () => {
|
||||
assert.match(pageSource, /getPrivilegedIdentityOutboxOverview/);
|
||||
assert.match(pageSource, /getPrivilegedIdentityOutboxDeadLetters/);
|
||||
assert.match(pageSource, /全局投递健康/);
|
||||
assert.match(pageSource, /死信排查/);
|
||||
assert.match(pageSource, /稳定错误码筛选/);
|
||||
assert.match(pageSource, /outboxOverview\.status_counts\.queued/);
|
||||
assert.match(pageSource, /outboxOverview\.expired_processing_lease_count/);
|
||||
assert.match(pageSource, /outboxOverview\.near_attempt_limit_count/);
|
||||
assert.match(pageSource, /outboxOverview\.dead_letter_count/);
|
||||
assert.match(pageSource, /deadLetters\.total_pages/);
|
||||
assert.match(pageSource, /外部 transport 与身份执行器保持关闭/);
|
||||
assert.doesNotMatch(
|
||||
pageSource,
|
||||
/item\.(payload|idempotency_key|lease_owner|last_error_detail|target_reference_fingerprint)/,
|
||||
);
|
||||
assert.doesNotMatch(pageSource, /retryPrivileged|requeuePrivileged|ackPrivileged|\/retry|\/requeue|\/ack/);
|
||||
});
|
||||
|
||||
test("outbox observability client calls are GET-only and keep filters in the query string", () => {
|
||||
const overviewFunction = clientSource.match(
|
||||
/export async function getPrivilegedIdentityOutboxOverview\([\s\S]*?\n\}/,
|
||||
)?.[0];
|
||||
const deadLettersFunction = clientSource.match(
|
||||
/export async function getPrivilegedIdentityOutboxDeadLetters\([\s\S]*?\n\}/,
|
||||
)?.[0];
|
||||
assert.ok(overviewFunction);
|
||||
assert.ok(deadLettersFunction);
|
||||
assert.match(overviewFunction, /privileged-outbox\/overview\?stale_after_seconds=/);
|
||||
assert.match(overviewFunction, /fetchClientJson/);
|
||||
assert.match(deadLettersFunction, /privileged-outbox\/dead-letters\?/);
|
||||
assert.match(deadLettersFunction, /page_size/);
|
||||
assert.match(deadLettersFunction, /error_code/);
|
||||
assert.match(deadLettersFunction, /fetchClientJson/);
|
||||
assert.doesNotMatch(`${overviewFunction}\n${deadLettersFunction}`, /method:\s*["']POST["']/);
|
||||
});
|
||||
|
||||
test("approval comments are required for both decisions", () => {
|
||||
|
||||
+290
-7
@@ -6,6 +6,8 @@ import {
|
||||
ArrowLeft,
|
||||
Activity,
|
||||
Check,
|
||||
ChevronLeft,
|
||||
ChevronRight,
|
||||
Eye,
|
||||
LockKeyhole,
|
||||
RefreshCw,
|
||||
@@ -16,11 +18,17 @@ import {
|
||||
createPrivilegedIdentityOperation,
|
||||
decidePrivilegedIdentityOperation,
|
||||
getPrivilegedIdentityOperation,
|
||||
getPrivilegedIdentityOperationDeliveryStatus,
|
||||
getPrivilegedIdentityOperationEvents,
|
||||
getPrivilegedIdentityOperations,
|
||||
getPrivilegedIdentityOutboxDeadLetters,
|
||||
getPrivilegedIdentityOutboxOverview,
|
||||
previewPrivilegedIdentityOperation,
|
||||
} from "@/services/contracts-client";
|
||||
import type {
|
||||
PrivilegedIdentityDeliveryStatus,
|
||||
PrivilegedIdentityOutboxDeadLetterPage,
|
||||
PrivilegedIdentityOutboxOverview,
|
||||
PrivilegedIdentityOperation,
|
||||
PrivilegedIdentityOperationEvent,
|
||||
PrivilegedIdentityOperationType,
|
||||
@@ -35,6 +43,8 @@ import {
|
||||
} from "@/lib/privileged-identity-operation-state";
|
||||
|
||||
const operationTypes = Object.keys(privilegedOperationDefinitions) as PrivilegedIdentityOperationType[];
|
||||
const outboxStaleAfterSeconds = 900;
|
||||
const deadLetterPageSize = 20;
|
||||
|
||||
export default function PrivilegedIdentityOperationsPage() {
|
||||
const [operationType, setOperationType] = useState<PrivilegedIdentityOperationType>("account_merge");
|
||||
@@ -48,11 +58,19 @@ export default function PrivilegedIdentityOperationsPage() {
|
||||
const [operations, setOperations] = useState<PrivilegedIdentityOperation[]>([]);
|
||||
const [selected, setSelected] = useState<PrivilegedIdentityOperation | null>(null);
|
||||
const [events, setEvents] = useState<PrivilegedIdentityOperationEvent[]>([]);
|
||||
const [deliveryStatus, setDeliveryStatus] = useState<PrivilegedIdentityDeliveryStatus[]>([]);
|
||||
const [eventsLoading, setEventsLoading] = useState(false);
|
||||
const [approvalComment, setApprovalComment] = useState("");
|
||||
const [message, setMessage] = useState<string | null>(null);
|
||||
const [busy, setBusy] = useState<string | null>(null);
|
||||
const [loading, setLoading] = useState(true);
|
||||
const [outboxOverview, setOutboxOverview] = useState<PrivilegedIdentityOutboxOverview | null>(null);
|
||||
const [overviewLoading, setOverviewLoading] = useState(true);
|
||||
const [overviewUnavailable, setOverviewUnavailable] = useState(false);
|
||||
const [deadLetters, setDeadLetters] = useState<PrivilegedIdentityOutboxDeadLetterPage | null>(null);
|
||||
const [deadLettersLoading, setDeadLettersLoading] = useState(true);
|
||||
const [deadLettersUnavailable, setDeadLettersUnavailable] = useState(false);
|
||||
const [deadLetterErrorCode, setDeadLetterErrorCode] = useState("");
|
||||
|
||||
const definition = privilegedOperationDefinitions[operationType];
|
||||
const targetRefs = useMemo(
|
||||
@@ -61,21 +79,62 @@ export default function PrivilegedIdentityOperationsPage() {
|
||||
);
|
||||
|
||||
useEffect(() => {
|
||||
void refreshOperations();
|
||||
void refreshGovernanceData();
|
||||
}, []);
|
||||
|
||||
async function refreshGovernanceData() {
|
||||
await Promise.all([
|
||||
refreshOperations(),
|
||||
refreshOutboxOverview(),
|
||||
refreshDeadLetters(deadLetters?.page ?? 1, deadLetterErrorCode),
|
||||
]);
|
||||
}
|
||||
|
||||
async function refreshOutboxOverview() {
|
||||
setOverviewLoading(true);
|
||||
try {
|
||||
const result = await getPrivilegedIdentityOutboxOverview(outboxStaleAfterSeconds);
|
||||
setOutboxOverview(result);
|
||||
setOverviewUnavailable(result === null);
|
||||
} finally {
|
||||
setOverviewLoading(false);
|
||||
}
|
||||
}
|
||||
|
||||
async function refreshDeadLetters(page: number, errorCode: string) {
|
||||
setDeadLettersLoading(true);
|
||||
try {
|
||||
const result = await getPrivilegedIdentityOutboxDeadLetters(
|
||||
page,
|
||||
deadLetterPageSize,
|
||||
errorCode,
|
||||
);
|
||||
setDeadLetters(result);
|
||||
setDeadLettersUnavailable(result === null);
|
||||
} finally {
|
||||
setDeadLettersLoading(false);
|
||||
}
|
||||
}
|
||||
|
||||
function filterDeadLetters(errorCode: string) {
|
||||
setDeadLetterErrorCode(errorCode);
|
||||
void refreshDeadLetters(1, errorCode);
|
||||
}
|
||||
|
||||
async function refreshOperations() {
|
||||
setLoading(true);
|
||||
try {
|
||||
const page = await getPrivilegedIdentityOperations();
|
||||
setOperations(page?.items ?? []);
|
||||
if (selected) {
|
||||
const [refreshed, eventPage] = await Promise.all([
|
||||
const [refreshed, eventPage, deliveryPage] = await Promise.all([
|
||||
getPrivilegedIdentityOperation(selected.operation_id),
|
||||
getPrivilegedIdentityOperationEvents(selected.operation_id),
|
||||
getPrivilegedIdentityOperationDeliveryStatus(selected.operation_id),
|
||||
]);
|
||||
setSelected(refreshed);
|
||||
setEvents(eventPage?.items ?? []);
|
||||
setDeliveryStatus(deliveryPage?.items ?? []);
|
||||
}
|
||||
} finally {
|
||||
setLoading(false);
|
||||
@@ -136,12 +195,14 @@ export default function PrivilegedIdentityOperationsPage() {
|
||||
setSelected(created);
|
||||
const eventPage = await getPrivilegedIdentityOperationEvents(created.operation_id);
|
||||
setEvents(eventPage?.items ?? []);
|
||||
const deliveryPage = await getPrivilegedIdentityOperationDeliveryStatus(created.operation_id);
|
||||
setDeliveryStatus(deliveryPage?.items ?? []);
|
||||
setPreview(null);
|
||||
setReason("");
|
||||
setTicketReference("");
|
||||
setRequestedExpiresAt("");
|
||||
setMessage("治理操作已登记，等待两名独立平台管理员审批。");
|
||||
await refreshOperations();
|
||||
await refreshGovernanceData();
|
||||
} catch (error) {
|
||||
setMessage(error instanceof Error ? error.message : "治理操作登记失败。");
|
||||
} finally {
|
||||
@@ -153,12 +214,14 @@ export default function PrivilegedIdentityOperationsPage() {
|
||||
setBusy(`load:${operation.operation_id}`);
|
||||
setEventsLoading(true);
|
||||
try {
|
||||
const [detail, eventPage] = await Promise.all([
|
||||
const [detail, eventPage, deliveryPage] = await Promise.all([
|
||||
getPrivilegedIdentityOperation(operation.operation_id),
|
||||
getPrivilegedIdentityOperationEvents(operation.operation_id),
|
||||
getPrivilegedIdentityOperationDeliveryStatus(operation.operation_id),
|
||||
]);
|
||||
setSelected(detail);
|
||||
setEvents(eventPage?.items ?? []);
|
||||
setDeliveryStatus(deliveryPage?.items ?? []);
|
||||
setApprovalComment("");
|
||||
} finally {
|
||||
setEventsLoading(false);
|
||||
@@ -179,7 +242,7 @@ export default function PrivilegedIdentityOperationsPage() {
|
||||
setSelected(updated);
|
||||
setApprovalComment("");
|
||||
setMessage(decision === "approve" ? "审批意见已记录。" : "拒绝意见已记录。");
|
||||
await refreshOperations();
|
||||
await refreshGovernanceData();
|
||||
} catch (error) {
|
||||
setMessage(error instanceof Error ? error.message : "审批提交失败。");
|
||||
} finally {
|
||||
@@ -206,6 +269,148 @@ export default function PrivilegedIdentityOperationsPage() {
|
||||
</div>
|
||||
{message && <div className="empty-state identity-message">{message}</div>}
|
||||
|
||||
<section className="panel">
|
||||
<div className="panel-header">
|
||||
<div>
|
||||
<p className="eyebrow">Outbox observability</p>
|
||||
<h2 className="panel-title">全局投递健康</h2>
|
||||
</div>
|
||||
{overviewLoading ? (
|
||||
<span className="tag tag-primary">读取中</span>
|
||||
) : outboxOverview ? (
|
||||
<span className={`tag ${outboxHealthTone(outboxOverview.health_status)}`}>
|
||||
{outboxHealthLabel(outboxOverview.health_status)}
|
||||
</span>
|
||||
) : (
|
||||
<span className="tag tag-danger">不可用</span>
|
||||
)}
|
||||
</div>
|
||||
<div className="panel-body list identity-outbox-summary">
|
||||
<div className="list-item">
|
||||
<span>
|
||||
<strong>Transport</strong>
|
||||
<span className="muted">外部 transport 与身份执行器保持关闭，本页只读观察，不会 claim、重试或修改消息。</span>
|
||||
</span>
|
||||
<span className="tag">Disabled</span>
|
||||
</div>
|
||||
{overviewLoading && <div className="empty-state">正在读取投递健康摘要...</div>}
|
||||
{!overviewLoading && overviewUnavailable && (
|
||||
<div className="empty-state empty-state-error">暂时无法读取全局投递健康摘要。</div>
|
||||
)}
|
||||
{!overviewLoading && outboxOverview && (
|
||||
<>
|
||||
<div className="list-item">
|
||||
<span><strong>消息状态</strong><span className="muted">queued · processing · retry · delivered · dead letter</span></span>
|
||||
<span className="toolbar toolbar-inline identity-outbox-statuses">
|
||||
<span className="tag">待投递 {outboxOverview.status_counts.queued}</span>
|
||||
<span className="tag tag-primary">投递中 {outboxOverview.status_counts.processing}</span>
|
||||
<span className="tag tag-warning">待重试 {outboxOverview.status_counts.retry_scheduled}</span>
|
||||
<span className="tag tag-success">已投递 {outboxOverview.status_counts.delivered}</span>
|
||||
<span className="tag tag-danger">死信 {outboxOverview.status_counts.dead_letter}</span>
|
||||
</span>
|
||||
</div>
|
||||
<div className="identity-outbox-metrics">
|
||||
<div className="list-item"><span>活动消息</span><strong>{outboxOverview.active_count}</strong></div>
|
||||
<div className="list-item"><span>最老活动消息</span><strong>{formatAge(outboxOverview.oldest_active_age_seconds)}</strong></div>
|
||||
<div className="list-item"><span>超过 {outboxOverview.stale_after_seconds} 秒阈值</span><strong>{outboxOverview.stale_active_count}</strong></div>
|
||||
<div className="list-item"><span>过期 processing lease</span><strong>{outboxOverview.expired_processing_lease_count}</strong></div>
|
||||
<div className="list-item"><span>接近最大尝试次数</span><strong>{outboxOverview.near_attempt_limit_count}</strong></div>
|
||||
<div className="list-item"><span>死信消息</span><strong>{outboxOverview.dead_letter_count}</strong></div>
|
||||
</div>
|
||||
<div className="list-item">
|
||||
<span><strong>死信错误码</strong><span className="muted">按稳定错误码聚合</span></span>
|
||||
<span className="toolbar toolbar-inline identity-outbox-statuses">
|
||||
{outboxOverview.dead_letter_error_codes.length === 0
|
||||
? <span className="muted">无</span>
|
||||
: outboxOverview.dead_letter_error_codes.map((item) => (
|
||||
<span className="tag tag-danger" key={item.error_code}>{item.error_code} · {item.count}</span>
|
||||
))}
|
||||
</span>
|
||||
</div>
|
||||
<div className="list-item"><span>摘要生成时间</span><strong>{formatDate(outboxOverview.generated_at)}</strong></div>
|
||||
</>
|
||||
)}
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section className="panel">
|
||||
<div className="panel-header">
|
||||
<div>
|
||||
<p className="eyebrow">Dead-letter investigation</p>
|
||||
<h2 className="panel-title">死信排查</h2>
|
||||
</div>
|
||||
<Activity size={20} />
|
||||
</div>
|
||||
<div className="panel-body">
|
||||
<div className="identity-form">
|
||||
<label>
|
||||
<span>稳定错误码筛选</span>
|
||||
<select
|
||||
aria-label="稳定错误码筛选"
|
||||
onChange={(event) => filterDeadLetters(event.target.value)}
|
||||
value={deadLetterErrorCode}
|
||||
>
|
||||
<option value="">全部错误码</option>
|
||||
{outboxOverview?.dead_letter_error_codes.map((item) => (
|
||||
<option key={item.error_code} value={item.error_code}>{item.error_code} ({item.count})</option>
|
||||
))}
|
||||
{deadLetterErrorCode && !outboxOverview?.dead_letter_error_codes.some((item) => item.error_code === deadLetterErrorCode) && (
|
||||
<option value={deadLetterErrorCode}>{deadLetterErrorCode}</option>
|
||||
)}
|
||||
</select>
|
||||
</label>
|
||||
</div>
|
||||
</div>
|
||||
<div className="panel-body list">
|
||||
{deadLettersLoading && <div className="empty-state">正在读取死信消息...</div>}
|
||||
{!deadLettersLoading && deadLettersUnavailable && (
|
||||
<div className="empty-state empty-state-error">暂时无法读取死信排查数据。</div>
|
||||
)}
|
||||
{!deadLettersLoading && !deadLettersUnavailable && deadLetters?.items.length === 0 && (
|
||||
<div className="empty-state">暂无死信消息。外部 transport 保持禁用，不会自动发送或重试。</div>
|
||||
)}
|
||||
{!deadLettersLoading && !deadLettersUnavailable && deadLetters?.items.map((item) => (
|
||||
<div className="list-item identity-event-item" key={item.message_id}>
|
||||
<span className="identity-event-copy">
|
||||
<strong>{deadLetterEventLabel(item.event_type)} · {item.effect_type}</strong>
|
||||
<span className="muted">消息 {item.message_id} · 事件 {item.event_id} · 操作 {item.operation_id}</span>
|
||||
<span className="muted">{privilegedOperationDefinitions[item.operation_type]?.label ?? item.operation_type} · trace_id: {item.trace_id || "未提供"}</span>
|
||||
<span className="muted">尝试 {item.attempt_count}/{item.max_attempts} · 创建于 {formatDate(item.created_at)} · 更新于 {formatDate(item.updated_at)}</span>
|
||||
{item.last_attempt_at && <span className="muted">最近尝试 {formatDate(item.last_attempt_at)}</span>}
|
||||
</span>
|
||||
<span className="tag tag-danger">{item.error_code}</span>
|
||||
</div>
|
||||
))}
|
||||
{!deadLettersLoading && !deadLettersUnavailable && deadLetters && deadLetters.total > 0 && (
|
||||
<div className="list-item">
|
||||
<span className="muted">第 {deadLetters.page} / {Math.max(deadLetters.total_pages, 1)} 页，共 {deadLetters.total} 条</span>
|
||||
<span className="toolbar toolbar-inline">
|
||||
<button
|
||||
aria-label="上一页"
|
||||
className="button button-soft"
|
||||
disabled={deadLetters.page <= 1 || deadLettersLoading}
|
||||
onClick={() => void refreshDeadLetters(deadLetters.page - 1, deadLetterErrorCode)}
|
||||
title="上一页"
|
||||
type="button"
|
||||
>
|
||||
<ChevronLeft size={16} />
|
||||
</button>
|
||||
<button
|
||||
aria-label="下一页"
|
||||
className="button button-soft"
|
||||
disabled={deadLetters.page >= deadLetters.total_pages || deadLettersLoading}
|
||||
onClick={() => void refreshDeadLetters(deadLetters.page + 1, deadLetterErrorCode)}
|
||||
title="下一页"
|
||||
type="button"
|
||||
>
|
||||
<ChevronRight size={16} />
|
||||
</button>
|
||||
</span>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<div className="identity-layout">
|
||||
<section className="panel">
|
||||
<div className="panel-header">
|
||||
@@ -333,8 +538,8 @@ export default function PrivilegedIdentityOperationsPage() {
|
||||
<p className="eyebrow">Operations</p>
|
||||
<h2 className="panel-title">治理操作记录</h2>
|
||||
</div>
|
||||
<button className="button button-soft" disabled={loading || busy !== null} onClick={() => void refreshOperations()} type="button">
|
||||
<RefreshCw size={16} /> 刷新
|
||||
<button className="button button-soft" disabled={loading || busy !== null} onClick={() => void refreshGovernanceData()} type="button">
|
||||
<RefreshCw size={16} /> 刷新全部
|
||||
</button>
|
||||
</div>
|
||||
<div className="panel-body list">
|
||||
@@ -426,6 +631,35 @@ export default function PrivilegedIdentityOperationsPage() {
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
<div className="panel-body list">
|
||||
<div className="list-item-header">
|
||||
<div>
|
||||
<p className="eyebrow">事件投递</p>
|
||||
<h3 className="panel-title">投递状态</h3>
|
||||
</div>
|
||||
<Activity size={18} />
|
||||
</div>
|
||||
{deliveryStatus.length === 0 && (
|
||||
<div className="empty-state">暂无投递状态记录。</div>
|
||||
)}
|
||||
{deliveryStatus.map((delivery) => (
|
||||
<div className="list-item identity-event-item" key={delivery.message_id}>
|
||||
<span className="identity-event-copy">
|
||||
<strong>{eventLabel(delivery.event_type)}</strong>
|
||||
<span className="muted">{delivery.message_id}</span>
|
||||
<span className="muted">
|
||||
尝试 {delivery.attempt_count}/{delivery.max_attempts} · 更新于 {formatDate(delivery.updated_at)}
|
||||
</span>
|
||||
{delivery.last_error_code && (
|
||||
<span className="muted">错误码 {delivery.last_error_code}</span>
|
||||
)}
|
||||
</span>
|
||||
<span className={`tag ${deliveryTone(delivery.status)}`}>
|
||||
{deliveryStatusLabel(delivery.status)}
|
||||
</span>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
{selected.status === "pending_approval" && (
|
||||
<div className="panel-body identity-form">
|
||||
<label>
|
||||
@@ -480,3 +714,52 @@ function snapshotSummary(snapshot: PrivilegedIdentityOperationEvent["snapshot_af
|
||||
`告警 ${snapshot.warning_count ?? 0}`,
|
||||
].join(" · ");
|
||||
}
|
||||
|
||||
function deliveryStatusLabel(status: PrivilegedIdentityDeliveryStatus["status"]) {
|
||||
return {
|
||||
queued: "待投递",
|
||||
processing: "投递中",
|
||||
retry_scheduled: "等待重试",
|
||||
delivered: "已投递",
|
||||
dead_letter: "需处理",
|
||||
}[status];
|
||||
}
|
||||
|
||||
function deliveryTone(status: PrivilegedIdentityDeliveryStatus["status"]) {
|
||||
if (status === "delivered") return "tag-success";
|
||||
if (status === "dead_letter") return "tag-danger";
|
||||
if (status === "processing") return "tag-primary";
|
||||
return "tag-warning";
|
||||
}
|
||||
|
||||
function outboxHealthLabel(status: PrivilegedIdentityOutboxOverview["health_status"]) {
|
||||
return {
|
||||
healthy: "正常",
|
||||
degraded: "告警",
|
||||
critical: "严重",
|
||||
}[status];
|
||||
}
|
||||
|
||||
function outboxHealthTone(status: PrivilegedIdentityOutboxOverview["health_status"]) {
|
||||
if (status === "healthy") return "tag-success";
|
||||
if (status === "critical") return "tag-danger";
|
||||
return "tag-warning";
|
||||
}
|
||||
|
||||
function formatAge(value?: number | null) {
|
||||
if (value === null || value === undefined) return "无活动消息";
|
||||
const seconds = Math.max(0, Math.floor(value));
|
||||
if (seconds < 60) return `${seconds} 秒`;
|
||||
if (seconds < 3600) return `${Math.floor(seconds / 60)} 分钟`;
|
||||
return `${(seconds / 3600).toFixed(seconds >= 36000 ? 0 : 1)} 小时`;
|
||||
}
|
||||
|
||||
function deadLetterEventLabel(eventType: string) {
|
||||
const labels: Record<string, string> = {
|
||||
operation_submitted: "登记操作",
|
||||
approval_recorded: "记录审批",
|
||||
operation_rejected: "拒绝操作",
|
||||
operation_expired: "操作过期",
|
||||
};
|
||||
return labels[eventType] ?? eventType;
|
||||
}
|
||||
|
||||
@@ -1252,6 +1252,45 @@ textarea {
|
||||
transform: translateY(-1px);
|
||||
}
|
||||
|
||||
.identity-outbox-summary .list-item {
|
||||
border: 0;
|
||||
border-bottom: 1px solid var(--line);
|
||||
border-radius: 0;
|
||||
background: transparent;
|
||||
box-shadow: none;
|
||||
transform: none;
|
||||
}
|
||||
|
||||
.identity-outbox-summary .list-item:last-child {
|
||||
border-bottom: 0;
|
||||
}
|
||||
|
||||
.identity-outbox-summary .list-item:hover {
|
||||
border-color: var(--line);
|
||||
background: rgba(255, 255, 255, 0.32);
|
||||
box-shadow: none;
|
||||
transform: none;
|
||||
}
|
||||
|
||||
.identity-outbox-metrics {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(2, minmax(0, 1fr));
|
||||
}
|
||||
|
||||
.identity-outbox-metrics .list-item:nth-child(odd) {
|
||||
border-right: 1px solid var(--line);
|
||||
}
|
||||
|
||||
@media (max-width: 760px) {
|
||||
.identity-outbox-metrics {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
.identity-outbox-metrics .list-item:nth-child(odd) {
|
||||
border-right: 0;
|
||||
}
|
||||
}
|
||||
|
||||
.list-item-header {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
|
||||
@@ -3,6 +3,9 @@ import type {
|
||||
LegalQaRequest,
|
||||
LegalQaResponse,
|
||||
PlatformAdministratorPage,
|
||||
PrivilegedIdentityDeliveryStatusPage,
|
||||
PrivilegedIdentityOutboxDeadLetterPage,
|
||||
PrivilegedIdentityOutboxOverview,
|
||||
PrivilegedIdentityOperation,
|
||||
PrivilegedIdentityOperationCreateRequest,
|
||||
PrivilegedIdentityOperationEventPage,
|
||||
@@ -83,6 +86,39 @@ export async function getPrivilegedIdentityOperationEvents(
|
||||
);
|
||||
}
|
||||
|
||||
export async function getPrivilegedIdentityOperationDeliveryStatus(
|
||||
operationId: string,
|
||||
): Promise<PrivilegedIdentityDeliveryStatusPage | null> {
|
||||
return fetchClientJson<PrivilegedIdentityDeliveryStatusPage>(
|
||||
`/api/contract-platform/identity/privileged-operations/${encodeURIComponent(operationId)}/delivery-status?page=1&page_size=100`,
|
||||
);
|
||||
}
|
||||
|
||||
export async function getPrivilegedIdentityOutboxOverview(
|
||||
staleAfterSeconds = 900,
|
||||
): Promise<PrivilegedIdentityOutboxOverview | null> {
|
||||
return fetchClientJson<PrivilegedIdentityOutboxOverview>(
|
||||
`/api/contract-platform/identity/privileged-outbox/overview?stale_after_seconds=${encodeURIComponent(String(staleAfterSeconds))}`,
|
||||
);
|
||||
}
|
||||
|
||||
export async function getPrivilegedIdentityOutboxDeadLetters(
|
||||
page = 1,
|
||||
pageSize = 20,
|
||||
errorCode = "",
|
||||
): Promise<PrivilegedIdentityOutboxDeadLetterPage | null> {
|
||||
const searchParams = new URLSearchParams({
|
||||
page: String(page),
|
||||
page_size: String(pageSize),
|
||||
});
|
||||
if (errorCode.trim()) {
|
||||
searchParams.set("error_code", errorCode.trim());
|
||||
}
|
||||
return fetchClientJson<PrivilegedIdentityOutboxDeadLetterPage>(
|
||||
`/api/contract-platform/identity/privileged-outbox/dead-letters?${searchParams.toString()}`,
|
||||
);
|
||||
}
|
||||
|
||||
export async function previewPrivilegedIdentityOperation(
|
||||
payload: PrivilegedIdentityPreviewRequest,
|
||||
): Promise<PrivilegedIdentityPreview> {
|
||||
|
||||
@@ -517,6 +517,79 @@ export type PrivilegedIdentityOperationEventPage = {
|
||||
page_size: number;
|
||||
};
|
||||
|
||||
export type PrivilegedIdentityDeliveryStatus = {
|
||||
message_id: string;
|
||||
event_id: string;
|
||||
event_type: PrivilegedIdentityOperationEventType;
|
||||
effect_type: string;
|
||||
status: "queued" | "processing" | "retry_scheduled" | "delivered" | "dead_letter";
|
||||
attempt_count: number;
|
||||
max_attempts: number;
|
||||
available_at: string;
|
||||
lease_expires_at?: string | null;
|
||||
last_error_code?: string | null;
|
||||
last_attempt_at?: string | null;
|
||||
delivered_at?: string | null;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
};
|
||||
|
||||
export type PrivilegedIdentityDeliveryStatusPage = {
|
||||
items: PrivilegedIdentityDeliveryStatus[];
|
||||
total: number;
|
||||
page: number;
|
||||
page_size: number;
|
||||
};
|
||||
|
||||
export type PrivilegedIdentityOutboxHealthStatus = "healthy" | "degraded" | "critical";
|
||||
|
||||
export type PrivilegedIdentityOutboxOverview = {
|
||||
health_status: PrivilegedIdentityOutboxHealthStatus;
|
||||
stale_after_seconds: number;
|
||||
status_counts: {
|
||||
queued: number;
|
||||
processing: number;
|
||||
retry_scheduled: number;
|
||||
delivered: number;
|
||||
dead_letter: number;
|
||||
};
|
||||
active_count: number;
|
||||
oldest_active_age_seconds?: number | null;
|
||||
stale_active_count: number;
|
||||
expired_processing_lease_count: number;
|
||||
near_attempt_limit_count: number;
|
||||
dead_letter_count: number;
|
||||
dead_letter_error_codes: Array<{
|
||||
error_code: string;
|
||||
count: number;
|
||||
}>;
|
||||
generated_at: string;
|
||||
};
|
||||
|
||||
export type PrivilegedIdentityOutboxDeadLetter = {
|
||||
message_id: string;
|
||||
event_id: string;
|
||||
operation_id: string;
|
||||
event_type: string;
|
||||
operation_type: PrivilegedIdentityOperationType;
|
||||
effect_type: string;
|
||||
trace_id?: string | null;
|
||||
attempt_count: number;
|
||||
max_attempts: number;
|
||||
error_code: string;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
last_attempt_at?: string | null;
|
||||
};
|
||||
|
||||
export type PrivilegedIdentityOutboxDeadLetterPage = {
|
||||
items: PrivilegedIdentityOutboxDeadLetter[];
|
||||
total: number;
|
||||
page: number;
|
||||
page_size: number;
|
||||
total_pages: number;
|
||||
};
|
||||
|
||||
export type PrivilegedIdentityOperation = {
|
||||
operation_id: string;
|
||||
operation_type: PrivilegedIdentityOperationType;
|
||||
@@ -1658,6 +1731,35 @@ export async function getPrivilegedIdentityOperation(
|
||||
);
|
||||
}
|
||||
|
||||
export async function getPrivilegedIdentityOutboxOverview(
|
||||
staleAfterSeconds = 900,
|
||||
): Promise<PrivilegedIdentityOutboxOverview | null> {
|
||||
const suffix = `?stale_after_seconds=${encodeURIComponent(String(staleAfterSeconds))}`;
|
||||
return fetchJson<PrivilegedIdentityOutboxOverview>(
|
||||
`/api/v1/identity/privileged-outbox/overview${suffix}`,
|
||||
`/api/contract-platform/identity/privileged-outbox/overview${suffix}`,
|
||||
);
|
||||
}
|
||||
|
||||
export async function getPrivilegedIdentityOutboxDeadLetters(
|
||||
page = 1,
|
||||
pageSize = 20,
|
||||
errorCode = "",
|
||||
): Promise<PrivilegedIdentityOutboxDeadLetterPage | null> {
|
||||
const searchParams = new URLSearchParams({
|
||||
page: String(page),
|
||||
page_size: String(pageSize),
|
||||
});
|
||||
if (errorCode.trim()) {
|
||||
searchParams.set("error_code", errorCode.trim());
|
||||
}
|
||||
const suffix = `?${searchParams.toString()}`;
|
||||
return fetchJson<PrivilegedIdentityOutboxDeadLetterPage>(
|
||||
`/api/v1/identity/privileged-outbox/dead-letters${suffix}`,
|
||||
`/api/contract-platform/identity/privileged-outbox/dead-letters${suffix}`,
|
||||
);
|
||||
}
|
||||
|
||||
export async function previewPrivilegedIdentityOperation(
|
||||
payload: PrivilegedIdentityPreviewRequest,
|
||||
): Promise<PrivilegedIdentityPreview> {
|
||||
|
||||
+121
@@ -0,0 +1,121 @@
|
||||
"""add privileged identity transactional outbox
|
||||
|
||||
Revision ID: 20260802_0040
|
||||
Revises: 20260802_0039
|
||||
Create Date: 2026-08-02
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision: str = "20260802_0040"
|
||||
down_revision: Union[str, None] = "20260802_0039"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"privileged_identity_outbox_messages",
|
||||
sa.Column("message_id", sa.String(length=64), nullable=False),
|
||||
sa.Column("operation_id", sa.String(length=64), nullable=False),
|
||||
sa.Column("event_id", sa.String(length=64), nullable=False),
|
||||
sa.Column("effect_type", sa.String(length=64), nullable=False),
|
||||
sa.Column("schema_version", sa.Integer(), nullable=False),
|
||||
sa.Column("idempotency_key", sa.String(length=160), nullable=False),
|
||||
sa.Column("payload", sa.JSON(), nullable=False),
|
||||
sa.Column("payload_hash", sa.String(length=64), nullable=False),
|
||||
sa.Column("status", sa.String(length=32), nullable=False),
|
||||
sa.Column("attempt_count", sa.Integer(), nullable=False),
|
||||
sa.Column("max_attempts", sa.Integer(), nullable=False),
|
||||
sa.Column("available_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("lease_owner", sa.String(length=128), nullable=True),
|
||||
sa.Column("lease_expires_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("last_error_code", sa.String(length=128), nullable=True),
|
||||
sa.Column("last_attempt_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("delivered_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.CheckConstraint(
|
||||
"status in ('queued', 'processing', 'retry_scheduled', 'delivered', 'dead_letter')",
|
||||
name="ck_privileged_identity_outbox_status",
|
||||
),
|
||||
sa.CheckConstraint(
|
||||
"attempt_count >= 0",
|
||||
name="ck_privileged_identity_outbox_attempt_count",
|
||||
),
|
||||
sa.CheckConstraint(
|
||||
"max_attempts > 0",
|
||||
name="ck_privileged_identity_outbox_max_attempts",
|
||||
),
|
||||
sa.CheckConstraint(
|
||||
"attempt_count <= max_attempts",
|
||||
name="ck_privileged_identity_outbox_attempt_limit",
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["operation_id"],
|
||||
["privileged_identity_operations.operation_id"],
|
||||
ondelete="NO ACTION",
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["event_id"],
|
||||
["privileged_identity_operation_events.event_id"],
|
||||
ondelete="NO ACTION",
|
||||
),
|
||||
sa.PrimaryKeyConstraint("message_id"),
|
||||
sa.UniqueConstraint(
|
||||
"event_id",
|
||||
"effect_type",
|
||||
name="ux_privileged_identity_outbox_event_effect",
|
||||
),
|
||||
sa.UniqueConstraint(
|
||||
"idempotency_key",
|
||||
name="ux_privileged_identity_outbox_idempotency_key",
|
||||
),
|
||||
)
|
||||
op.create_index(
|
||||
"ix_privileged_identity_outbox_dispatch",
|
||||
"privileged_identity_outbox_messages",
|
||||
["status", "available_at", "created_at"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_privileged_identity_outbox_operation_created",
|
||||
"privileged_identity_outbox_messages",
|
||||
["operation_id", "created_at"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_privileged_identity_outbox_status_lease",
|
||||
"privileged_identity_outbox_messages",
|
||||
["status", "lease_expires_at"],
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
undelivered_count = op.get_bind().scalar(
|
||||
sa.text(
|
||||
"SELECT COUNT(*) FROM privileged_identity_outbox_messages "
|
||||
"WHERE status <> 'delivered'"
|
||||
)
|
||||
)
|
||||
if int(undelivered_count or 0) > 0:
|
||||
raise RuntimeError(
|
||||
"Cannot downgrade privileged identity outbox while undelivered messages remain."
|
||||
)
|
||||
op.drop_index(
|
||||
"ix_privileged_identity_outbox_status_lease",
|
||||
table_name="privileged_identity_outbox_messages",
|
||||
)
|
||||
op.drop_index(
|
||||
"ix_privileged_identity_outbox_operation_created",
|
||||
table_name="privileged_identity_outbox_messages",
|
||||
)
|
||||
op.drop_index(
|
||||
"ix_privileged_identity_outbox_dispatch",
|
||||
table_name="privileged_identity_outbox_messages",
|
||||
)
|
||||
op.drop_table("privileged_identity_outbox_messages")
|
||||
+180
@@ -0,0 +1,180 @@
|
||||
"""enforce privileged identity outbox integrity
|
||||
|
||||
Revision ID: 20260803_0041
|
||||
Revises: 20260802_0040
|
||||
Create Date: 2026-08-03
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision: str = "20260803_0041"
|
||||
down_revision: Union[str, None] = "20260802_0040"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
_EVENT_OPERATION_INDEX = "ux_privileged_identity_operation_events_operation_event"
|
||||
_EVENT_OPERATION_FOREIGN_KEY = "fk_privileged_identity_outbox_event_operation"
|
||||
_LEASE_STATE_CHECK = "ck_privileged_identity_outbox_lease_state"
|
||||
_SQLITE_INSERT_TRIGGER = "trg_privileged_identity_outbox_event_operation_insert"
|
||||
_SQLITE_UPDATE_TRIGGER = "trg_privileged_identity_outbox_event_operation_update"
|
||||
|
||||
|
||||
def _create_sqlite_event_operation_triggers() -> None:
|
||||
op.execute(
|
||||
sa.text(
|
||||
f"""
|
||||
CREATE TRIGGER {_SQLITE_INSERT_TRIGGER}
|
||||
BEFORE INSERT ON privileged_identity_outbox_messages
|
||||
WHEN NOT EXISTS (
|
||||
SELECT 1
|
||||
FROM privileged_identity_operation_events AS event
|
||||
WHERE event.operation_id = NEW.operation_id
|
||||
AND event.event_id = NEW.event_id
|
||||
)
|
||||
BEGIN
|
||||
SELECT RAISE(ABORT, 'privileged identity outbox event/operation mismatch');
|
||||
END
|
||||
"""
|
||||
)
|
||||
)
|
||||
op.execute(
|
||||
sa.text(
|
||||
f"""
|
||||
CREATE TRIGGER {_SQLITE_UPDATE_TRIGGER}
|
||||
BEFORE UPDATE OF operation_id, event_id ON privileged_identity_outbox_messages
|
||||
WHEN NOT EXISTS (
|
||||
SELECT 1
|
||||
FROM privileged_identity_operation_events AS event
|
||||
WHERE event.operation_id = NEW.operation_id
|
||||
AND event.event_id = NEW.event_id
|
||||
)
|
||||
BEGIN
|
||||
SELECT RAISE(ABORT, 'privileged identity outbox event/operation mismatch');
|
||||
END
|
||||
"""
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _assert_existing_rows_are_valid() -> None:
|
||||
bind = op.get_bind()
|
||||
mismatched_event_count = bind.scalar(
|
||||
sa.text(
|
||||
"""
|
||||
SELECT COUNT(*)
|
||||
FROM privileged_identity_outbox_messages AS outbox
|
||||
LEFT JOIN privileged_identity_operation_events AS event
|
||||
ON event.event_id = outbox.event_id
|
||||
WHERE event.event_id IS NULL
|
||||
OR event.operation_id <> outbox.operation_id
|
||||
"""
|
||||
)
|
||||
)
|
||||
invalid_lease_count = bind.scalar(
|
||||
sa.text(
|
||||
"""
|
||||
SELECT COUNT(*)
|
||||
FROM privileged_identity_outbox_messages
|
||||
WHERE (
|
||||
status = 'processing'
|
||||
AND (lease_owner IS NULL OR lease_expires_at IS NULL)
|
||||
) OR (
|
||||
status <> 'processing'
|
||||
AND (lease_owner IS NOT NULL OR lease_expires_at IS NOT NULL)
|
||||
)
|
||||
"""
|
||||
)
|
||||
)
|
||||
if int(mismatched_event_count or 0) > 0:
|
||||
raise RuntimeError(
|
||||
"Cannot enforce privileged identity outbox integrity while event/operation "
|
||||
"mismatches exist. Investigate the audit linkage before retrying the migration."
|
||||
)
|
||||
if int(invalid_lease_count or 0) > 0:
|
||||
raise RuntimeError(
|
||||
"Cannot enforce privileged identity outbox integrity while invalid lease state "
|
||||
"rows exist. Investigate the worker state before retrying the migration."
|
||||
)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
_assert_existing_rows_are_valid()
|
||||
op.create_index(
|
||||
_EVENT_OPERATION_INDEX,
|
||||
"privileged_identity_operation_events",
|
||||
["operation_id", "event_id"],
|
||||
unique=True,
|
||||
)
|
||||
lease_state_expression = (
|
||||
"(status = 'processing' AND lease_owner IS NOT NULL "
|
||||
"AND lease_expires_at IS NOT NULL) OR "
|
||||
"(status <> 'processing' AND lease_owner IS NULL "
|
||||
"AND lease_expires_at IS NULL)"
|
||||
)
|
||||
bind = op.get_bind()
|
||||
if bind.dialect.name == "sqlite":
|
||||
with op.batch_alter_table(
|
||||
"privileged_identity_outbox_messages",
|
||||
recreate="always",
|
||||
) as batch_op:
|
||||
batch_op.create_foreign_key(
|
||||
_EVENT_OPERATION_FOREIGN_KEY,
|
||||
"privileged_identity_operation_events",
|
||||
["operation_id", "event_id"],
|
||||
["operation_id", "event_id"],
|
||||
ondelete="NO ACTION",
|
||||
)
|
||||
batch_op.create_check_constraint(
|
||||
_LEASE_STATE_CHECK,
|
||||
lease_state_expression,
|
||||
)
|
||||
_create_sqlite_event_operation_triggers()
|
||||
else:
|
||||
op.create_foreign_key(
|
||||
_EVENT_OPERATION_FOREIGN_KEY,
|
||||
"privileged_identity_outbox_messages",
|
||||
"privileged_identity_operation_events",
|
||||
["operation_id", "event_id"],
|
||||
["operation_id", "event_id"],
|
||||
ondelete="NO ACTION",
|
||||
)
|
||||
op.create_check_constraint(
|
||||
_LEASE_STATE_CHECK,
|
||||
"privileged_identity_outbox_messages",
|
||||
lease_state_expression,
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
bind = op.get_bind()
|
||||
if bind.dialect.name == "sqlite":
|
||||
op.execute(sa.text(f"DROP TRIGGER IF EXISTS {_SQLITE_UPDATE_TRIGGER}"))
|
||||
op.execute(sa.text(f"DROP TRIGGER IF EXISTS {_SQLITE_INSERT_TRIGGER}"))
|
||||
with op.batch_alter_table(
|
||||
"privileged_identity_outbox_messages",
|
||||
recreate="always",
|
||||
) as batch_op:
|
||||
batch_op.drop_constraint(_EVENT_OPERATION_FOREIGN_KEY, type_="foreignkey")
|
||||
batch_op.drop_constraint(_LEASE_STATE_CHECK, type_="check")
|
||||
else:
|
||||
op.drop_constraint(
|
||||
_EVENT_OPERATION_FOREIGN_KEY,
|
||||
"privileged_identity_outbox_messages",
|
||||
type_="foreignkey",
|
||||
)
|
||||
op.drop_constraint(
|
||||
_LEASE_STATE_CHECK,
|
||||
"privileged_identity_outbox_messages",
|
||||
type_="check",
|
||||
)
|
||||
op.drop_index(
|
||||
_EVENT_OPERATION_INDEX,
|
||||
table_name="privileged_identity_operation_events",
|
||||
)
|
||||
@@ -621,6 +621,17 @@ def contract_endpoint_risk_policies(api_prefix: str) -> tuple[EndpointRiskPolicy
|
||||
"records",
|
||||
True,
|
||||
),
|
||||
EndpointRiskPolicy(
|
||||
"identity.privileged_outbox.read",
|
||||
"GET",
|
||||
f"{p}/identity/privileged-outbox",
|
||||
"S3",
|
||||
20,
|
||||
100,
|
||||
40,
|
||||
"records",
|
||||
True,
|
||||
),
|
||||
EndpointRiskPolicy("audit", "GET", f"{p}/audit", "S2", 20, 200, 60, "records", True),
|
||||
)
|
||||
|
||||
|
||||
@@ -62,6 +62,7 @@ from yuqei_contract_api.identity import (
|
||||
)
|
||||
from yuqei_contract_api.persistence.models import DEFAULT_TENANT_ID
|
||||
from yuqei_contract_api.privileged_identity import (
|
||||
PrivilegedIdentityDeliveryStatusPage,
|
||||
PrivilegedIdentityOperationApprovalRequest,
|
||||
PrivilegedIdentityOperationError,
|
||||
PrivilegedIdentityOperationEventPage,
|
||||
@@ -72,6 +73,8 @@ from yuqei_contract_api.privileged_identity import (
|
||||
PrivilegedIdentityOperationStatus,
|
||||
PrivilegedIdentityOperationSubmitRequest,
|
||||
PrivilegedIdentityOperationType,
|
||||
PrivilegedIdentityOutboxDeadLetterPage,
|
||||
PrivilegedIdentityOutboxOverview,
|
||||
)
|
||||
from yuqei_contract_api.logging import get_logger
|
||||
from yuqei_contract_api.repository_base import ContractRepository
|
||||
@@ -286,6 +289,71 @@ def register_identity_routes(
|
||||
except PrivilegedIdentityOperationError as exc:
|
||||
raise _privileged_operation_http_error(exc) from exc
|
||||
|
||||
@app.get(
|
||||
f"{settings.api_prefix}/identity/privileged-operations/{{operation_id}}/delivery-status",
|
||||
response_model=PrivilegedIdentityDeliveryStatusPage,
|
||||
tags=["identity"],
|
||||
)
|
||||
def list_privileged_identity_operation_delivery_status(
|
||||
operation_id: str,
|
||||
page: int = Query(default=1, ge=1),
|
||||
page_size: int = Query(default=50, ge=1, le=100),
|
||||
) -> PrivilegedIdentityDeliveryStatusPage:
|
||||
_require_privileged_platform_admin(settings, identity_repository)
|
||||
privileged_repository = _privileged_operation_repository(identity_repository)
|
||||
try:
|
||||
return privileged_repository.list_delivery_status(
|
||||
operation_id,
|
||||
actor_tenant_id=_current_tenant_id(),
|
||||
actor_user_id=_current_user_id(),
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
)
|
||||
except PrivilegedIdentityOperationError as exc:
|
||||
raise _privileged_operation_http_error(exc) from exc
|
||||
|
||||
@app.get(
|
||||
f"{settings.api_prefix}/identity/privileged-outbox/overview",
|
||||
response_model=PrivilegedIdentityOutboxOverview,
|
||||
tags=["identity"],
|
||||
)
|
||||
def get_privileged_identity_outbox_overview(
|
||||
stale_after_seconds: int = Query(default=900, ge=60, le=86_400),
|
||||
) -> PrivilegedIdentityOutboxOverview:
|
||||
_require_privileged_platform_admin(settings, identity_repository)
|
||||
privileged_repository = _privileged_operation_repository(identity_repository)
|
||||
try:
|
||||
return privileged_repository.get_outbox_overview(
|
||||
actor_tenant_id=_current_tenant_id(),
|
||||
actor_user_id=_current_user_id(),
|
||||
stale_after_seconds=stale_after_seconds,
|
||||
)
|
||||
except PrivilegedIdentityOperationError as exc:
|
||||
raise _privileged_operation_http_error(exc) from exc
|
||||
|
||||
@app.get(
|
||||
f"{settings.api_prefix}/identity/privileged-outbox/dead-letters",
|
||||
response_model=PrivilegedIdentityOutboxDeadLetterPage,
|
||||
tags=["identity"],
|
||||
)
|
||||
def list_privileged_identity_outbox_dead_letters(
|
||||
page: int = Query(default=1, ge=1),
|
||||
page_size: int = Query(default=20, ge=1, le=100),
|
||||
error_code: str | None = Query(default=None, min_length=1, max_length=128),
|
||||
) -> PrivilegedIdentityOutboxDeadLetterPage:
|
||||
_require_privileged_platform_admin(settings, identity_repository)
|
||||
privileged_repository = _privileged_operation_repository(identity_repository)
|
||||
try:
|
||||
return privileged_repository.list_outbox_dead_letters(
|
||||
actor_tenant_id=_current_tenant_id(),
|
||||
actor_user_id=_current_user_id(),
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
stable_error_code=error_code,
|
||||
)
|
||||
except PrivilegedIdentityOperationError as exc:
|
||||
raise _privileged_operation_http_error(exc) from exc
|
||||
|
||||
@app.post(
|
||||
f"{settings.api_prefix}/identity/privileged-operations/{{operation_id}}/approvals",
|
||||
response_model=PrivilegedIdentityOperationItem,
|
||||
|
||||
+44
-1
@@ -8,10 +8,13 @@ _SQLITE_UPDATE_TRIGGER = "trg_privileged_identity_operation_events_no_update"
|
||||
_SQLITE_DELETE_TRIGGER = "trg_privileged_identity_operation_events_no_delete"
|
||||
_POSTGRES_TRIGGER = "trg_privileged_identity_operation_events_immutable"
|
||||
_POSTGRES_FUNCTION = "privileged_identity_operation_events_immutable"
|
||||
_OUTBOX_TABLE = "privileged_identity_outbox_messages"
|
||||
_SQLITE_OUTBOX_INSERT_TRIGGER = "trg_privileged_identity_outbox_event_operation_insert"
|
||||
_SQLITE_OUTBOX_UPDATE_TRIGGER = "trg_privileged_identity_outbox_event_operation_update"
|
||||
|
||||
|
||||
def install_privileged_identity_operation_event_guards(engine: Engine) -> None:
|
||||
"""Prevent direct mutation of append-only governance events on auto-created databases."""
|
||||
"""Install governance integrity guards on auto-created databases."""
|
||||
dialect = engine.dialect.name
|
||||
if dialect == "sqlite":
|
||||
with engine.begin() as connection:
|
||||
@@ -26,6 +29,46 @@ def install_privileged_identity_operation_event_guards(engine: Engine) -> None:
|
||||
"""
|
||||
)
|
||||
)
|
||||
connection.execute(
|
||||
text(
|
||||
f"""
|
||||
CREATE TRIGGER IF NOT EXISTS {_SQLITE_OUTBOX_INSERT_TRIGGER}
|
||||
BEFORE INSERT ON {_OUTBOX_TABLE}
|
||||
WHEN NOT EXISTS (
|
||||
SELECT 1
|
||||
FROM {_EVENT_TABLE} AS event
|
||||
WHERE event.operation_id = NEW.operation_id
|
||||
AND event.event_id = NEW.event_id
|
||||
)
|
||||
BEGIN
|
||||
SELECT RAISE(
|
||||
ABORT,
|
||||
'privileged identity outbox event/operation mismatch'
|
||||
);
|
||||
END
|
||||
"""
|
||||
)
|
||||
)
|
||||
connection.execute(
|
||||
text(
|
||||
f"""
|
||||
CREATE TRIGGER IF NOT EXISTS {_SQLITE_OUTBOX_UPDATE_TRIGGER}
|
||||
BEFORE UPDATE OF operation_id, event_id ON {_OUTBOX_TABLE}
|
||||
WHEN NOT EXISTS (
|
||||
SELECT 1
|
||||
FROM {_EVENT_TABLE} AS event
|
||||
WHERE event.operation_id = NEW.operation_id
|
||||
AND event.event_id = NEW.event_id
|
||||
)
|
||||
BEGIN
|
||||
SELECT RAISE(
|
||||
ABORT,
|
||||
'privileged identity outbox event/operation mismatch'
|
||||
);
|
||||
END
|
||||
"""
|
||||
)
|
||||
)
|
||||
connection.execute(
|
||||
text(
|
||||
f"""
|
||||
|
||||
+98
@@ -9,6 +9,7 @@ from sqlalchemy import (
|
||||
DateTime,
|
||||
Float,
|
||||
ForeignKey,
|
||||
ForeignKeyConstraint,
|
||||
Index,
|
||||
Integer,
|
||||
JSON,
|
||||
@@ -289,6 +290,12 @@ class PrivilegedIdentityOperationEventModel(Base):
|
||||
"sequence_no",
|
||||
name="ux_privileged_identity_operation_events_operation_sequence",
|
||||
),
|
||||
Index(
|
||||
"ux_privileged_identity_operation_events_operation_event",
|
||||
"operation_id",
|
||||
"event_id",
|
||||
unique=True,
|
||||
),
|
||||
Index("ix_privileged_identity_operation_events_operation_created", "operation_id", "created_at"),
|
||||
Index("ix_privileged_identity_operation_events_type_created", "event_type", "created_at"),
|
||||
Index("ix_privileged_identity_operation_events_trace", "trace_id"),
|
||||
@@ -317,6 +324,97 @@ class PrivilegedIdentityOperationEventModel(Base):
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
|
||||
|
||||
class PrivilegedIdentityOutboxMessageModel(Base):
|
||||
__tablename__ = "privileged_identity_outbox_messages"
|
||||
__table_args__ = (
|
||||
CheckConstraint(
|
||||
"status in ('queued', 'processing', 'retry_scheduled', 'delivered', 'dead_letter')",
|
||||
name="ck_privileged_identity_outbox_status",
|
||||
),
|
||||
CheckConstraint(
|
||||
"attempt_count >= 0",
|
||||
name="ck_privileged_identity_outbox_attempt_count",
|
||||
),
|
||||
CheckConstraint(
|
||||
"max_attempts > 0",
|
||||
name="ck_privileged_identity_outbox_max_attempts",
|
||||
),
|
||||
CheckConstraint(
|
||||
"attempt_count <= max_attempts",
|
||||
name="ck_privileged_identity_outbox_attempt_limit",
|
||||
),
|
||||
CheckConstraint(
|
||||
"(status = 'processing' AND lease_owner IS NOT NULL "
|
||||
"AND lease_expires_at IS NOT NULL) OR "
|
||||
"(status <> 'processing' AND lease_owner IS NULL "
|
||||
"AND lease_expires_at IS NULL)",
|
||||
name="ck_privileged_identity_outbox_lease_state",
|
||||
),
|
||||
ForeignKeyConstraint(
|
||||
["operation_id", "event_id"],
|
||||
[
|
||||
"privileged_identity_operation_events.operation_id",
|
||||
"privileged_identity_operation_events.event_id",
|
||||
],
|
||||
name="fk_privileged_identity_outbox_event_operation",
|
||||
ondelete="NO ACTION",
|
||||
),
|
||||
UniqueConstraint(
|
||||
"event_id",
|
||||
"effect_type",
|
||||
name="ux_privileged_identity_outbox_event_effect",
|
||||
),
|
||||
UniqueConstraint(
|
||||
"idempotency_key",
|
||||
name="ux_privileged_identity_outbox_idempotency_key",
|
||||
),
|
||||
Index(
|
||||
"ix_privileged_identity_outbox_dispatch",
|
||||
"status",
|
||||
"available_at",
|
||||
"created_at",
|
||||
),
|
||||
Index(
|
||||
"ix_privileged_identity_outbox_operation_created",
|
||||
"operation_id",
|
||||
"created_at",
|
||||
),
|
||||
Index(
|
||||
"ix_privileged_identity_outbox_status_lease",
|
||||
"status",
|
||||
"lease_expires_at",
|
||||
),
|
||||
)
|
||||
|
||||
message_id: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||||
operation_id: Mapped[str] = mapped_column(
|
||||
String(64),
|
||||
ForeignKey("privileged_identity_operations.operation_id", ondelete="NO ACTION"),
|
||||
nullable=False,
|
||||
)
|
||||
event_id: Mapped[str] = mapped_column(
|
||||
String(64),
|
||||
ForeignKey("privileged_identity_operation_events.event_id", ondelete="NO ACTION"),
|
||||
nullable=False,
|
||||
)
|
||||
effect_type: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
schema_version: Mapped[int] = mapped_column(Integer, nullable=False, default=1)
|
||||
idempotency_key: Mapped[str] = mapped_column(String(160), nullable=False)
|
||||
payload: Mapped[dict] = mapped_column(JSON, nullable=False)
|
||||
payload_hash: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
status: Mapped[str] = mapped_column(String(32), nullable=False, default="queued")
|
||||
attempt_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
max_attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=5)
|
||||
available_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
lease_owner: Mapped[str | None] = mapped_column(String(128), nullable=True)
|
||||
lease_expires_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
last_error_code: Mapped[str | None] = mapped_column(String(128), nullable=True)
|
||||
last_attempt_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
delivered_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
|
||||
|
||||
class TenantInvitationModel(GovernanceMixin, Base):
|
||||
__tablename__ = "tenant_invitations"
|
||||
__table_args__ = (
|
||||
|
||||
+413
-14
@@ -18,14 +18,22 @@ from yuqei_contract_api.persistence.models import (
|
||||
PrivilegedIdentityOperationApprovalModel,
|
||||
PrivilegedIdentityOperationEventModel,
|
||||
PrivilegedIdentityOperationModel,
|
||||
PrivilegedIdentityOutboxMessageModel,
|
||||
TenantModel,
|
||||
TenantUserCredentialModel,
|
||||
TenantUserModel,
|
||||
WorkspaceMembershipModel,
|
||||
)
|
||||
from yuqei_contract_api.privileged_identity import (
|
||||
PrivilegedIdentityDeliveryStatusItem,
|
||||
PrivilegedIdentityDeliveryStatusPage,
|
||||
PrivilegedIdentityEvidenceRef,
|
||||
PrivilegedIdentityExpectedRevision,
|
||||
PrivilegedIdentityOutboxDeadLetterItem,
|
||||
PrivilegedIdentityOutboxDeadLetterPage,
|
||||
PrivilegedIdentityOutboxErrorCodeCount,
|
||||
PrivilegedIdentityOutboxOverview,
|
||||
PrivilegedIdentityOutboxStatusCounts,
|
||||
PrivilegedIdentityOperationApprovalItem,
|
||||
PrivilegedIdentityOperationApprovalRequest,
|
||||
PrivilegedIdentityOperationError,
|
||||
@@ -54,6 +62,9 @@ _SUPPORT_ACCESS_SCOPES = {
|
||||
"review.read",
|
||||
"template.read",
|
||||
}
|
||||
_OUTBOX_EFFECT_TYPE = "privileged_identity.operation_event"
|
||||
_OUTBOX_SCHEMA_VERSION = 1
|
||||
_OUTBOX_MAX_ATTEMPTS = 5
|
||||
_TARGET_REF_KEYS: dict[PrivilegedIdentityOperationType, set[str]] = {
|
||||
"account_merge": {"source_account_id", "target_account_id"},
|
||||
"account_split": {"source_account_id", "destination_account_id", "membership_partition"},
|
||||
@@ -320,6 +331,257 @@ class PrivilegedIdentityOperationRepository:
|
||||
page_size=page_size,
|
||||
)
|
||||
|
||||
def list_delivery_status(
|
||||
self,
|
||||
operation_id: str,
|
||||
*,
|
||||
actor_tenant_id: str,
|
||||
actor_user_id: str,
|
||||
page: int = 1,
|
||||
page_size: int = 50,
|
||||
) -> PrivilegedIdentityDeliveryStatusPage:
|
||||
with self._session_factory() as session:
|
||||
self._require_current_platform_admin(
|
||||
session,
|
||||
tenant_id=actor_tenant_id,
|
||||
user_id=actor_user_id,
|
||||
)
|
||||
operation = session.get(PrivilegedIdentityOperationModel, operation_id)
|
||||
if operation is None:
|
||||
raise _error(404, "operation_not_found", "Privileged identity operation not found.")
|
||||
total = int(
|
||||
session.execute(
|
||||
select(func.count(PrivilegedIdentityOutboxMessageModel.message_id)).where(
|
||||
PrivilegedIdentityOutboxMessageModel.operation_id == operation_id
|
||||
)
|
||||
).scalar_one()
|
||||
)
|
||||
rows = session.execute(
|
||||
select(
|
||||
PrivilegedIdentityOutboxMessageModel,
|
||||
PrivilegedIdentityOperationEventModel.event_type,
|
||||
)
|
||||
.join(
|
||||
PrivilegedIdentityOperationEventModel,
|
||||
PrivilegedIdentityOperationEventModel.event_id
|
||||
== PrivilegedIdentityOutboxMessageModel.event_id,
|
||||
)
|
||||
.where(PrivilegedIdentityOutboxMessageModel.operation_id == operation_id)
|
||||
.order_by(
|
||||
PrivilegedIdentityOperationEventModel.sequence_no,
|
||||
PrivilegedIdentityOutboxMessageModel.message_id,
|
||||
)
|
||||
.offset((page - 1) * page_size)
|
||||
.limit(page_size)
|
||||
).all()
|
||||
return PrivilegedIdentityDeliveryStatusPage(
|
||||
items=[
|
||||
_delivery_status_from_model(message, event_type=cast(str, event_type))
|
||||
for message, event_type in rows
|
||||
],
|
||||
total=total,
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
)
|
||||
|
||||
def get_outbox_overview(
|
||||
self,
|
||||
*,
|
||||
actor_tenant_id: str,
|
||||
actor_user_id: str,
|
||||
stale_after_seconds: int = 900,
|
||||
) -> PrivilegedIdentityOutboxOverview:
|
||||
if not 60 <= stale_after_seconds <= 86_400:
|
||||
raise _error(
|
||||
422,
|
||||
"invalid_stale_threshold",
|
||||
"Outbox stale threshold must be between 60 and 86400 seconds.",
|
||||
)
|
||||
observed_at = datetime.now(UTC)
|
||||
stale_before = observed_at - timedelta(seconds=stale_after_seconds)
|
||||
active_statuses = ("queued", "processing", "retry_scheduled")
|
||||
with self._session_factory() as session:
|
||||
self._require_current_platform_admin(
|
||||
session,
|
||||
tenant_id=actor_tenant_id,
|
||||
user_id=actor_user_id,
|
||||
)
|
||||
status_rows = session.execute(
|
||||
select(
|
||||
PrivilegedIdentityOutboxMessageModel.status,
|
||||
func.count(PrivilegedIdentityOutboxMessageModel.message_id),
|
||||
).group_by(PrivilegedIdentityOutboxMessageModel.status)
|
||||
).all()
|
||||
raw_status_counts = {str(status): int(count) for status, count in status_rows}
|
||||
status_counts = PrivilegedIdentityOutboxStatusCounts(
|
||||
**{
|
||||
status: raw_status_counts.get(status, 0)
|
||||
for status in PrivilegedIdentityOutboxStatusCounts.model_fields
|
||||
}
|
||||
)
|
||||
oldest_active_created_at = session.execute(
|
||||
select(func.min(PrivilegedIdentityOutboxMessageModel.created_at)).where(
|
||||
PrivilegedIdentityOutboxMessageModel.status.in_(active_statuses)
|
||||
)
|
||||
).scalar_one()
|
||||
stale_active_count = _count_outbox_messages(
|
||||
session,
|
||||
PrivilegedIdentityOutboxMessageModel.status.in_(active_statuses),
|
||||
PrivilegedIdentityOutboxMessageModel.created_at <= stale_before,
|
||||
)
|
||||
expired_processing_lease_count = _count_outbox_messages(
|
||||
session,
|
||||
PrivilegedIdentityOutboxMessageModel.status == "processing",
|
||||
PrivilegedIdentityOutboxMessageModel.lease_expires_at.is_not(None),
|
||||
PrivilegedIdentityOutboxMessageModel.lease_expires_at <= observed_at,
|
||||
)
|
||||
near_attempt_limit_count = _count_outbox_messages(
|
||||
session,
|
||||
PrivilegedIdentityOutboxMessageModel.status.in_(active_statuses),
|
||||
PrivilegedIdentityOutboxMessageModel.attempt_count
|
||||
>= PrivilegedIdentityOutboxMessageModel.max_attempts - 1,
|
||||
)
|
||||
stable_error_code_expression = func.coalesce(
|
||||
PrivilegedIdentityOutboxMessageModel.last_error_code,
|
||||
"unknown",
|
||||
)
|
||||
dead_letter_error_rows = session.execute(
|
||||
select(
|
||||
stable_error_code_expression,
|
||||
func.count(PrivilegedIdentityOutboxMessageModel.message_id),
|
||||
)
|
||||
.where(PrivilegedIdentityOutboxMessageModel.status == "dead_letter")
|
||||
.group_by(stable_error_code_expression)
|
||||
.order_by(
|
||||
func.count(PrivilegedIdentityOutboxMessageModel.message_id).desc(),
|
||||
stable_error_code_expression,
|
||||
)
|
||||
).all()
|
||||
|
||||
active_count = (
|
||||
status_counts.queued
|
||||
+ status_counts.processing
|
||||
+ status_counts.retry_scheduled
|
||||
)
|
||||
oldest_active_age_seconds = None
|
||||
if oldest_active_created_at is not None:
|
||||
oldest_active_age_seconds = max(
|
||||
0,
|
||||
int((observed_at - _as_utc(oldest_active_created_at)).total_seconds()),
|
||||
)
|
||||
if status_counts.dead_letter > 0 or expired_processing_lease_count > 0:
|
||||
health = "critical"
|
||||
elif stale_active_count > 0 or near_attempt_limit_count > 0:
|
||||
health = "degraded"
|
||||
else:
|
||||
health = "healthy"
|
||||
return PrivilegedIdentityOutboxOverview(
|
||||
health_status=cast(str, health),
|
||||
generated_at=observed_at,
|
||||
stale_after_seconds=stale_after_seconds,
|
||||
status_counts=status_counts,
|
||||
active_count=active_count,
|
||||
oldest_active_age_seconds=oldest_active_age_seconds,
|
||||
stale_active_count=stale_active_count,
|
||||
expired_processing_lease_count=expired_processing_lease_count,
|
||||
near_attempt_limit_count=near_attempt_limit_count,
|
||||
dead_letter_count=status_counts.dead_letter,
|
||||
dead_letter_error_codes=[
|
||||
PrivilegedIdentityOutboxErrorCodeCount(
|
||||
error_code=error_code,
|
||||
count=int(count),
|
||||
)
|
||||
for error_code, count in dead_letter_error_rows
|
||||
],
|
||||
)
|
||||
|
||||
def list_outbox_dead_letters(
|
||||
self,
|
||||
*,
|
||||
actor_tenant_id: str,
|
||||
actor_user_id: str,
|
||||
page: int = 1,
|
||||
page_size: int = 20,
|
||||
stable_error_code: str | None = None,
|
||||
) -> PrivilegedIdentityOutboxDeadLetterPage:
|
||||
normalized_error_code = None
|
||||
if stable_error_code is not None:
|
||||
normalized_error_code = stable_error_code.strip().lower()
|
||||
if not re.fullmatch(r"[a-z][a-z0-9_.:-]{0,127}", normalized_error_code):
|
||||
raise _error(
|
||||
422,
|
||||
"invalid_stable_error_code",
|
||||
"Stable outbox error code is invalid.",
|
||||
)
|
||||
with self._session_factory() as session:
|
||||
self._require_current_platform_admin(
|
||||
session,
|
||||
tenant_id=actor_tenant_id,
|
||||
user_id=actor_user_id,
|
||||
)
|
||||
filters = [PrivilegedIdentityOutboxMessageModel.status == "dead_letter"]
|
||||
if normalized_error_code is not None:
|
||||
filters.append(
|
||||
PrivilegedIdentityOutboxMessageModel.last_error_code.is_(None)
|
||||
if normalized_error_code == "unknown"
|
||||
else PrivilegedIdentityOutboxMessageModel.last_error_code
|
||||
== normalized_error_code
|
||||
)
|
||||
total = _count_outbox_messages(session, *filters)
|
||||
rows = session.execute(
|
||||
select(
|
||||
PrivilegedIdentityOutboxMessageModel,
|
||||
PrivilegedIdentityOperationEventModel.event_type,
|
||||
PrivilegedIdentityOperationEventModel.trace_id,
|
||||
PrivilegedIdentityOperationModel.operation_type,
|
||||
)
|
||||
.join(
|
||||
PrivilegedIdentityOperationEventModel,
|
||||
PrivilegedIdentityOperationEventModel.event_id
|
||||
== PrivilegedIdentityOutboxMessageModel.event_id,
|
||||
)
|
||||
.join(
|
||||
PrivilegedIdentityOperationModel,
|
||||
PrivilegedIdentityOperationModel.operation_id
|
||||
== PrivilegedIdentityOutboxMessageModel.operation_id,
|
||||
)
|
||||
.where(*filters)
|
||||
.order_by(
|
||||
PrivilegedIdentityOutboxMessageModel.updated_at.desc(),
|
||||
PrivilegedIdentityOutboxMessageModel.message_id.desc(),
|
||||
)
|
||||
.offset((page - 1) * page_size)
|
||||
.limit(page_size)
|
||||
).all()
|
||||
return PrivilegedIdentityOutboxDeadLetterPage(
|
||||
items=[
|
||||
PrivilegedIdentityOutboxDeadLetterItem(
|
||||
message_id=message.message_id,
|
||||
event_id=message.event_id,
|
||||
operation_id=message.operation_id,
|
||||
event_type=cast(PrivilegedIdentityOperationEventType, event_type),
|
||||
operation_type=cast(PrivilegedIdentityOperationType, operation_type),
|
||||
effect_type=message.effect_type,
|
||||
trace_id=_safe_trace_id(trace_id),
|
||||
attempt_count=message.attempt_count,
|
||||
max_attempts=message.max_attempts,
|
||||
error_code=message.last_error_code or "unknown",
|
||||
created_at=_as_utc(message.created_at),
|
||||
updated_at=_as_utc(message.updated_at),
|
||||
last_attempt_at=(
|
||||
_as_utc(message.last_attempt_at)
|
||||
if message.last_attempt_at
|
||||
else None
|
||||
),
|
||||
)
|
||||
for message, event_type, trace_id, operation_type in rows
|
||||
],
|
||||
total=total,
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
total_pages=(total + page_size - 1) // page_size if total else 0,
|
||||
)
|
||||
|
||||
def decide(
|
||||
self,
|
||||
operation_id: str,
|
||||
@@ -613,23 +875,101 @@ class PrivilegedIdentityOperationRepository:
|
||||
PrivilegedIdentityOperationEventModel.operation_id == operation.operation_id
|
||||
)
|
||||
).scalar_one()
|
||||
sequence_no = int(current_sequence or 0) + 1
|
||||
event_id = f"privileged-event-{uuid4().hex[:24]}"
|
||||
trace_id = _safe_trace_id(get_trace_id())
|
||||
before_hash = privileged_payload_hash(snapshot_before)
|
||||
after_hash = privileged_payload_hash(snapshot_after)
|
||||
event = PrivilegedIdentityOperationEventModel(
|
||||
event_id=event_id,
|
||||
operation_id=operation.operation_id,
|
||||
sequence_no=sequence_no,
|
||||
event_type=event_type,
|
||||
status_before=status_before,
|
||||
status_after=status_after,
|
||||
trace_id=trace_id,
|
||||
actor_account_id=actor_account_id,
|
||||
actor_tenant_id=actor_tenant_id,
|
||||
actor_user_id=actor_user_id,
|
||||
snapshot_before=snapshot_before,
|
||||
snapshot_after=snapshot_after,
|
||||
snapshot_before_hash=before_hash,
|
||||
snapshot_after_hash=after_hash,
|
||||
created_at=now,
|
||||
)
|
||||
session.add(event)
|
||||
session.flush()
|
||||
self._append_outbox_message(
|
||||
session,
|
||||
operation=operation,
|
||||
event=event,
|
||||
snapshot_after=snapshot_after,
|
||||
now=now,
|
||||
)
|
||||
|
||||
def _append_outbox_message(
|
||||
self,
|
||||
session: Session,
|
||||
*,
|
||||
operation: PrivilegedIdentityOperationModel,
|
||||
event: PrivilegedIdentityOperationEventModel,
|
||||
snapshot_after: dict[str, object],
|
||||
now: datetime,
|
||||
) -> None:
|
||||
payload: dict[str, object] = {
|
||||
"event_id": event.event_id,
|
||||
"operation_id": operation.operation_id,
|
||||
"sequence_no": event.sequence_no,
|
||||
"event_type": event.event_type,
|
||||
"operation_type": operation.operation_type,
|
||||
"status_before": event.status_before,
|
||||
"status_after": event.status_after,
|
||||
"approval_count": int(snapshot_after.get("approval_count", 0)),
|
||||
"target_reference_count": int(
|
||||
snapshot_after.get("target_reference_count", 0)
|
||||
),
|
||||
"target_reference_fingerprints": list(
|
||||
snapshot_after.get("target_reference_fingerprints", [])
|
||||
),
|
||||
"evidence_reference_count": int(
|
||||
snapshot_after.get("evidence_reference_count", 0)
|
||||
),
|
||||
"warning_count": int(snapshot_after.get("warning_count", 0)),
|
||||
"trace_id": event.trace_id,
|
||||
"snapshot_before_hash": event.snapshot_before_hash,
|
||||
"snapshot_after_hash": event.snapshot_after_hash,
|
||||
"occurred_at": _as_utc(now).isoformat(),
|
||||
}
|
||||
_validate_outbox_payload(payload)
|
||||
idempotency_key = "privileged-identity:" + privileged_payload_hash(
|
||||
{
|
||||
"operation_id": operation.operation_id,
|
||||
"sequence_no": event.sequence_no,
|
||||
"effect_type": _OUTBOX_EFFECT_TYPE,
|
||||
"schema_version": _OUTBOX_SCHEMA_VERSION,
|
||||
}
|
||||
)
|
||||
session.add(
|
||||
PrivilegedIdentityOperationEventModel(
|
||||
event_id=f"privileged-event-{uuid4().hex[:24]}",
|
||||
PrivilegedIdentityOutboxMessageModel(
|
||||
message_id=f"privileged-outbox-{uuid4().hex[:24]}",
|
||||
operation_id=operation.operation_id,
|
||||
sequence_no=int(current_sequence or 0) + 1,
|
||||
event_type=event_type,
|
||||
status_before=status_before,
|
||||
status_after=status_after,
|
||||
trace_id=_safe_trace_id(get_trace_id()),
|
||||
actor_account_id=actor_account_id,
|
||||
actor_tenant_id=actor_tenant_id,
|
||||
actor_user_id=actor_user_id,
|
||||
snapshot_before=snapshot_before,
|
||||
snapshot_after=snapshot_after,
|
||||
snapshot_before_hash=privileged_payload_hash(snapshot_before),
|
||||
snapshot_after_hash=privileged_payload_hash(snapshot_after),
|
||||
event_id=event.event_id,
|
||||
effect_type=_OUTBOX_EFFECT_TYPE,
|
||||
schema_version=_OUTBOX_SCHEMA_VERSION,
|
||||
idempotency_key=idempotency_key,
|
||||
payload=payload,
|
||||
payload_hash=privileged_payload_hash(payload),
|
||||
status="queued",
|
||||
attempt_count=0,
|
||||
max_attempts=_OUTBOX_MAX_ATTEMPTS,
|
||||
available_at=now,
|
||||
lease_owner=None,
|
||||
lease_expires_at=None,
|
||||
last_error_code=None,
|
||||
last_attempt_at=None,
|
||||
delivered_at=None,
|
||||
created_at=now,
|
||||
updated_at=now,
|
||||
)
|
||||
)
|
||||
|
||||
@@ -1281,5 +1621,64 @@ def _event_from_model(model: PrivilegedIdentityOperationEventModel) -> Privilege
|
||||
)
|
||||
|
||||
|
||||
def _delivery_status_from_model(
|
||||
model: PrivilegedIdentityOutboxMessageModel,
|
||||
*,
|
||||
event_type: str,
|
||||
) -> PrivilegedIdentityDeliveryStatusItem:
|
||||
return PrivilegedIdentityDeliveryStatusItem(
|
||||
message_id=model.message_id,
|
||||
event_id=model.event_id,
|
||||
event_type=cast(PrivilegedIdentityOperationEventType, event_type),
|
||||
effect_type=model.effect_type,
|
||||
status=cast(str, model.status),
|
||||
attempt_count=model.attempt_count,
|
||||
max_attempts=model.max_attempts,
|
||||
available_at=_as_utc(model.available_at),
|
||||
lease_expires_at=_as_utc(model.lease_expires_at) if model.lease_expires_at else None,
|
||||
last_error_code=model.last_error_code,
|
||||
last_attempt_at=_as_utc(model.last_attempt_at) if model.last_attempt_at else None,
|
||||
delivered_at=_as_utc(model.delivered_at) if model.delivered_at else None,
|
||||
created_at=_as_utc(model.created_at),
|
||||
updated_at=_as_utc(model.updated_at),
|
||||
)
|
||||
|
||||
|
||||
def _validate_outbox_payload(payload: dict[str, object]) -> None:
|
||||
allowed_keys = {
|
||||
"event_id",
|
||||
"operation_id",
|
||||
"sequence_no",
|
||||
"event_type",
|
||||
"operation_type",
|
||||
"status_before",
|
||||
"status_after",
|
||||
"approval_count",
|
||||
"target_reference_count",
|
||||
"target_reference_fingerprints",
|
||||
"evidence_reference_count",
|
||||
"warning_count",
|
||||
"trace_id",
|
||||
"snapshot_before_hash",
|
||||
"snapshot_after_hash",
|
||||
"occurred_at",
|
||||
}
|
||||
if set(payload) != allowed_keys:
|
||||
raise ValueError("Privileged identity outbox payload does not match the allowlist.")
|
||||
fingerprints = payload["target_reference_fingerprints"]
|
||||
if not isinstance(fingerprints, list) or not all(
|
||||
isinstance(item, str) and re.fullmatch(r"[0-9a-f]{32}", item)
|
||||
for item in fingerprints
|
||||
):
|
||||
raise ValueError("Privileged identity outbox fingerprints are invalid.")
|
||||
|
||||
|
||||
def _error(status_code: int, code: str, message: str) -> PrivilegedIdentityOperationError:
|
||||
return PrivilegedIdentityOperationError(status_code=status_code, code=code, message=message)
|
||||
|
||||
|
||||
def _count_outbox_messages(session: Session, *conditions) -> int:
|
||||
statement = select(func.count(PrivilegedIdentityOutboxMessageModel.message_id))
|
||||
if conditions:
|
||||
statement = statement.where(*conditions)
|
||||
return int(session.execute(statement).scalar_one())
|
||||
|
||||
+292
@@ -0,0 +1,292 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime, timedelta
|
||||
import re
|
||||
from threading import Lock
|
||||
from typing import Protocol
|
||||
|
||||
from sqlalchemy import and_, or_, select, text, update
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from yuqei_contract_api.persistence.models import PrivilegedIdentityOutboxMessageModel
|
||||
from yuqei_contract_api.privileged_identity import privileged_payload_hash
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ClaimedPrivilegedIdentityMessage:
|
||||
message_id: str
|
||||
operation_id: str
|
||||
event_id: str
|
||||
effect_type: str
|
||||
schema_version: int
|
||||
idempotency_key: str
|
||||
payload: dict[str, object]
|
||||
payload_hash: str
|
||||
attempt_count: int
|
||||
max_attempts: int
|
||||
|
||||
|
||||
class PrivilegedIdentityOutboxRepository:
|
||||
"""Lease-based at-least-once outbox storage without a configured transport."""
|
||||
|
||||
def __init__(self, session_factory: sessionmaker) -> None:
|
||||
self._session_factory = session_factory
|
||||
|
||||
@property
|
||||
def dialect_name(self) -> str:
|
||||
with self._session_factory() as session:
|
||||
return session.get_bind().dialect.name
|
||||
|
||||
def claim_batch(
|
||||
self,
|
||||
*,
|
||||
worker_id: str,
|
||||
batch_size: int = 20,
|
||||
lease_seconds: int = 60,
|
||||
now: datetime | None = None,
|
||||
) -> list[ClaimedPrivilegedIdentityMessage]:
|
||||
if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._:-]{0,127}", worker_id):
|
||||
raise ValueError("Outbox worker_id is invalid.")
|
||||
if not 1 <= batch_size <= 100:
|
||||
raise ValueError("Outbox batch_size must be between 1 and 100.")
|
||||
if not 5 <= lease_seconds <= 900:
|
||||
raise ValueError("Outbox lease_seconds must be between 5 and 900.")
|
||||
current = _as_utc(now or datetime.now(UTC))
|
||||
lease_expiry = current + timedelta(seconds=lease_seconds)
|
||||
with self._session_factory.begin() as session:
|
||||
if session.get_bind().dialect.name == "sqlite":
|
||||
session.execute(text("BEGIN IMMEDIATE"))
|
||||
session.execute(
|
||||
update(PrivilegedIdentityOutboxMessageModel)
|
||||
.where(
|
||||
PrivilegedIdentityOutboxMessageModel.status == "processing",
|
||||
PrivilegedIdentityOutboxMessageModel.lease_expires_at <= current,
|
||||
PrivilegedIdentityOutboxMessageModel.attempt_count
|
||||
>= PrivilegedIdentityOutboxMessageModel.max_attempts,
|
||||
)
|
||||
.values(
|
||||
status="dead_letter",
|
||||
lease_owner=None,
|
||||
lease_expires_at=None,
|
||||
last_error_code="lease_attempts_exhausted",
|
||||
updated_at=current,
|
||||
)
|
||||
)
|
||||
statement = (
|
||||
select(PrivilegedIdentityOutboxMessageModel)
|
||||
.where(
|
||||
PrivilegedIdentityOutboxMessageModel.attempt_count
|
||||
< PrivilegedIdentityOutboxMessageModel.max_attempts,
|
||||
or_(
|
||||
and_(
|
||||
PrivilegedIdentityOutboxMessageModel.status.in_(
|
||||
["queued", "retry_scheduled"]
|
||||
),
|
||||
PrivilegedIdentityOutboxMessageModel.available_at <= current,
|
||||
),
|
||||
and_(
|
||||
PrivilegedIdentityOutboxMessageModel.status == "processing",
|
||||
PrivilegedIdentityOutboxMessageModel.lease_expires_at <= current,
|
||||
),
|
||||
),
|
||||
)
|
||||
.order_by(
|
||||
PrivilegedIdentityOutboxMessageModel.available_at,
|
||||
PrivilegedIdentityOutboxMessageModel.created_at,
|
||||
PrivilegedIdentityOutboxMessageModel.message_id,
|
||||
)
|
||||
.limit(batch_size)
|
||||
)
|
||||
if session.get_bind().dialect.name == "postgresql":
|
||||
statement = statement.with_for_update(skip_locked=True)
|
||||
models = session.execute(statement).scalars().all()
|
||||
claimed: list[ClaimedPrivilegedIdentityMessage] = []
|
||||
for model in models:
|
||||
model.status = "processing"
|
||||
model.attempt_count += 1
|
||||
model.lease_owner = worker_id
|
||||
model.lease_expires_at = lease_expiry
|
||||
model.last_attempt_at = current
|
||||
model.updated_at = current
|
||||
claimed.append(_claimed_message(model))
|
||||
session.flush()
|
||||
return claimed
|
||||
|
||||
def mark_delivered(
|
||||
self,
|
||||
message_id: str,
|
||||
*,
|
||||
worker_id: str,
|
||||
now: datetime | None = None,
|
||||
) -> bool:
|
||||
current = _as_utc(now or datetime.now(UTC))
|
||||
with self._session_factory.begin() as session:
|
||||
result = session.execute(
|
||||
update(PrivilegedIdentityOutboxMessageModel)
|
||||
.where(
|
||||
PrivilegedIdentityOutboxMessageModel.message_id == message_id,
|
||||
PrivilegedIdentityOutboxMessageModel.status == "processing",
|
||||
PrivilegedIdentityOutboxMessageModel.lease_owner == worker_id,
|
||||
PrivilegedIdentityOutboxMessageModel.lease_expires_at > current,
|
||||
)
|
||||
.values(
|
||||
status="delivered",
|
||||
lease_owner=None,
|
||||
lease_expires_at=None,
|
||||
last_error_code=None,
|
||||
delivered_at=current,
|
||||
updated_at=current,
|
||||
)
|
||||
)
|
||||
return bool(result.rowcount)
|
||||
|
||||
def mark_failed(
|
||||
self,
|
||||
message_id: str,
|
||||
*,
|
||||
worker_id: str,
|
||||
error_code: str,
|
||||
now: datetime | None = None,
|
||||
permanent: bool = False,
|
||||
) -> bool:
|
||||
current = _as_utc(now or datetime.now(UTC))
|
||||
safe_error = _safe_error_code(error_code)
|
||||
with self._session_factory.begin() as session:
|
||||
model = session.execute(
|
||||
select(PrivilegedIdentityOutboxMessageModel)
|
||||
.where(
|
||||
PrivilegedIdentityOutboxMessageModel.message_id == message_id,
|
||||
PrivilegedIdentityOutboxMessageModel.status == "processing",
|
||||
PrivilegedIdentityOutboxMessageModel.lease_owner == worker_id,
|
||||
PrivilegedIdentityOutboxMessageModel.lease_expires_at > current,
|
||||
)
|
||||
.with_for_update()
|
||||
).scalar_one_or_none()
|
||||
if model is None:
|
||||
return False
|
||||
exhausted = permanent or model.attempt_count >= model.max_attempts
|
||||
model.status = "dead_letter" if exhausted else "retry_scheduled"
|
||||
model.available_at = current + _retry_delay(model.attempt_count)
|
||||
model.lease_owner = None
|
||||
model.lease_expires_at = None
|
||||
model.last_error_code = safe_error
|
||||
model.updated_at = current
|
||||
return True
|
||||
|
||||
|
||||
_SQLITE_WORKER_LOCK = Lock()
|
||||
|
||||
|
||||
class PrivilegedIdentityOutboxTransport(Protocol):
|
||||
enabled: bool
|
||||
|
||||
def deliver(self, message: ClaimedPrivilegedIdentityMessage) -> None: ...
|
||||
|
||||
|
||||
class DisabledPrivilegedIdentityOutboxTransport:
|
||||
enabled = False
|
||||
|
||||
def deliver(self, message: ClaimedPrivilegedIdentityMessage) -> None:
|
||||
raise RuntimeError("Privileged identity outbox transport is disabled.")
|
||||
|
||||
|
||||
class PrivilegedIdentityOutboxWorker:
|
||||
"""Transport-neutral worker; SQLite execution is intentionally single-worker."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
repository: PrivilegedIdentityOutboxRepository,
|
||||
*,
|
||||
worker_id: str,
|
||||
transport: PrivilegedIdentityOutboxTransport | None = None,
|
||||
) -> None:
|
||||
self._repository = repository
|
||||
self._worker_id = worker_id
|
||||
self._transport = transport or DisabledPrivilegedIdentityOutboxTransport()
|
||||
|
||||
def run_once(
|
||||
self,
|
||||
deliver: Callable[[ClaimedPrivilegedIdentityMessage], None] | None = None,
|
||||
*,
|
||||
batch_size: int = 20,
|
||||
lease_seconds: int = 60,
|
||||
now: datetime | None = None,
|
||||
) -> int:
|
||||
if deliver is None and not self._transport.enabled:
|
||||
return 0
|
||||
delivery = deliver or self._transport.deliver
|
||||
current = _as_utc(now or datetime.now(UTC))
|
||||
sqlite_lock_acquired = False
|
||||
if self._repository.dialect_name == "sqlite":
|
||||
sqlite_lock_acquired = _SQLITE_WORKER_LOCK.acquire(blocking=False)
|
||||
if not sqlite_lock_acquired:
|
||||
raise RuntimeError("SQLite privileged identity outbox supports one worker only.")
|
||||
try:
|
||||
messages = self._repository.claim_batch(
|
||||
worker_id=self._worker_id,
|
||||
batch_size=batch_size,
|
||||
lease_seconds=lease_seconds,
|
||||
now=current,
|
||||
)
|
||||
delivered = 0
|
||||
for message in messages:
|
||||
if privileged_payload_hash(message.payload) != message.payload_hash:
|
||||
self._repository.mark_failed(
|
||||
message.message_id,
|
||||
worker_id=self._worker_id,
|
||||
error_code="payload_hash_mismatch",
|
||||
permanent=True,
|
||||
)
|
||||
continue
|
||||
try:
|
||||
delivery(message)
|
||||
except Exception: # Transport failures remain retryable and payload-free.
|
||||
self._repository.mark_failed(
|
||||
message.message_id,
|
||||
worker_id=self._worker_id,
|
||||
error_code="transport_error",
|
||||
)
|
||||
continue
|
||||
if self._repository.mark_delivered(
|
||||
message.message_id,
|
||||
worker_id=self._worker_id,
|
||||
):
|
||||
delivered += 1
|
||||
return delivered
|
||||
finally:
|
||||
if sqlite_lock_acquired:
|
||||
_SQLITE_WORKER_LOCK.release()
|
||||
|
||||
|
||||
def _claimed_message(model: PrivilegedIdentityOutboxMessageModel) -> ClaimedPrivilegedIdentityMessage:
|
||||
return ClaimedPrivilegedIdentityMessage(
|
||||
message_id=model.message_id,
|
||||
operation_id=model.operation_id,
|
||||
event_id=model.event_id,
|
||||
effect_type=model.effect_type,
|
||||
schema_version=model.schema_version,
|
||||
idempotency_key=model.idempotency_key,
|
||||
payload=dict(model.payload or {}),
|
||||
payload_hash=model.payload_hash,
|
||||
attempt_count=model.attempt_count,
|
||||
max_attempts=model.max_attempts,
|
||||
)
|
||||
|
||||
|
||||
def _retry_delay(attempt_count: int) -> timedelta:
|
||||
return timedelta(seconds=min(300, 2 ** min(max(attempt_count, 1), 8)))
|
||||
|
||||
|
||||
def _safe_error_code(value: str) -> str:
|
||||
candidate = value.strip().lower()[:128]
|
||||
if re.fullmatch(r"[a-z][a-z0-9_.:-]{0,127}", candidate):
|
||||
return candidate
|
||||
return "transport_error"
|
||||
|
||||
|
||||
def _as_utc(value: datetime) -> datetime:
|
||||
if value.tzinfo is None:
|
||||
return value.replace(tzinfo=UTC)
|
||||
return value.astimezone(UTC)
|
||||
+82
@@ -30,6 +30,13 @@ PrivilegedIdentityOperationEventType = Literal[
|
||||
"operation_rejected",
|
||||
"operation_expired",
|
||||
]
|
||||
PrivilegedIdentityOutboxStatus = Literal[
|
||||
"queued",
|
||||
"processing",
|
||||
"retry_scheduled",
|
||||
"delivered",
|
||||
"dead_letter",
|
||||
]
|
||||
|
||||
|
||||
class PrivilegedIdentityOperationError(ValueError):
|
||||
@@ -156,6 +163,81 @@ class PrivilegedIdentityOperationEventPage(BaseModel):
|
||||
page_size: int
|
||||
|
||||
|
||||
class PrivilegedIdentityDeliveryStatusItem(BaseModel):
|
||||
message_id: str
|
||||
event_id: str
|
||||
event_type: PrivilegedIdentityOperationEventType
|
||||
effect_type: str
|
||||
status: PrivilegedIdentityOutboxStatus
|
||||
attempt_count: int
|
||||
max_attempts: int
|
||||
available_at: datetime
|
||||
lease_expires_at: datetime | None = None
|
||||
last_error_code: str | None = None
|
||||
last_attempt_at: datetime | None = None
|
||||
delivered_at: datetime | None = None
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
|
||||
|
||||
class PrivilegedIdentityDeliveryStatusPage(BaseModel):
|
||||
items: list[PrivilegedIdentityDeliveryStatusItem]
|
||||
total: int
|
||||
page: int
|
||||
page_size: int
|
||||
|
||||
|
||||
class PrivilegedIdentityOutboxStatusCounts(BaseModel):
|
||||
queued: int = 0
|
||||
processing: int = 0
|
||||
retry_scheduled: int = 0
|
||||
delivered: int = 0
|
||||
dead_letter: int = 0
|
||||
|
||||
|
||||
class PrivilegedIdentityOutboxErrorCodeCount(BaseModel):
|
||||
error_code: str
|
||||
count: int
|
||||
|
||||
|
||||
class PrivilegedIdentityOutboxOverview(BaseModel):
|
||||
health_status: Literal["healthy", "degraded", "critical"]
|
||||
generated_at: datetime
|
||||
stale_after_seconds: int
|
||||
status_counts: PrivilegedIdentityOutboxStatusCounts
|
||||
active_count: int
|
||||
oldest_active_age_seconds: int | None = None
|
||||
stale_active_count: int
|
||||
expired_processing_lease_count: int
|
||||
near_attempt_limit_count: int
|
||||
dead_letter_count: int
|
||||
dead_letter_error_codes: list[PrivilegedIdentityOutboxErrorCodeCount]
|
||||
|
||||
|
||||
class PrivilegedIdentityOutboxDeadLetterItem(BaseModel):
|
||||
message_id: str
|
||||
event_id: str
|
||||
operation_id: str
|
||||
event_type: PrivilegedIdentityOperationEventType
|
||||
operation_type: PrivilegedIdentityOperationType
|
||||
effect_type: str
|
||||
trace_id: str | None = None
|
||||
attempt_count: int
|
||||
max_attempts: int
|
||||
error_code: str
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
last_attempt_at: datetime | None = None
|
||||
|
||||
|
||||
class PrivilegedIdentityOutboxDeadLetterPage(BaseModel):
|
||||
items: list[PrivilegedIdentityOutboxDeadLetterItem]
|
||||
total: int
|
||||
page: int
|
||||
page_size: int
|
||||
total_pages: int
|
||||
|
||||
|
||||
def normalize_privileged_json(value: object) -> object:
|
||||
if value is None or isinstance(value, bool):
|
||||
return value
|
||||
|
||||
@@ -3,6 +3,7 @@ from datetime import UTC, date, datetime
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from yuqei_contract_api.anti_abuse import contract_endpoint_risk_policies, match_policy
|
||||
from yuqei_contract_api.config import ContractApiSettings
|
||||
from yuqei_contract_api.main import create_app
|
||||
from yuqei_contract_api.persistence.models import DataEgressLedgerModel, SecurityAccessEventModel
|
||||
@@ -20,6 +21,26 @@ def make_settings(**overrides) -> ContractApiSettings:
|
||||
return ContractApiSettings(**values)
|
||||
|
||||
|
||||
def test_privileged_outbox_read_routes_use_sensitive_s3_policy() -> None:
|
||||
policies = contract_endpoint_risk_policies("/api/v1")
|
||||
|
||||
for path in (
|
||||
"/api/v1/identity/privileged-outbox/overview",
|
||||
"/api/v1/identity/privileged-outbox/dead-letters",
|
||||
):
|
||||
policy = match_policy("GET", path, policies)
|
||||
assert policy is not None
|
||||
assert policy.route_key == "identity.privileged_outbox.read"
|
||||
assert policy.data_class == "S3"
|
||||
assert policy.sensitive is True
|
||||
|
||||
assert match_policy(
|
||||
"POST",
|
||||
"/api/v1/identity/privileged-outbox/dead-letters",
|
||||
policies,
|
||||
) is None
|
||||
|
||||
|
||||
def test_contract_anti_abuse_observe_marks_sensitive_responses() -> None:
|
||||
app = create_app(make_settings(anti_abuse_mode="observe"))
|
||||
client = TestClient(app)
|
||||
|
||||
+345
-1
@@ -1,8 +1,12 @@
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
from alembic import command
|
||||
from alembic.config import Config
|
||||
from sqlalchemy import create_engine, inspect
|
||||
import pytest
|
||||
from sqlalchemy import create_engine, inspect, text
|
||||
from sqlalchemy.engine import Connection
|
||||
from sqlalchemy.exc import DBAPIError
|
||||
|
||||
from yuqei_contract_api.config import get_settings
|
||||
|
||||
@@ -10,6 +14,66 @@ from yuqei_contract_api.config import get_settings
|
||||
SERVICE_ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def _insert_migration_operation_and_event(
|
||||
connection: Connection,
|
||||
*,
|
||||
suffix: str,
|
||||
now: str,
|
||||
) -> None:
|
||||
connection.execute(
|
||||
text(
|
||||
"""
|
||||
INSERT INTO privileged_identity_operations (
|
||||
operation_id, operation_type, status, request_payload_hash,
|
||||
requested_by_account_id, requested_by_tenant_id,
|
||||
requested_by_user_id, reason, ticket_reference, target_refs,
|
||||
evidence_refs, preview_hash, expected_revisions, impact_summary,
|
||||
warnings, expires_at, created_at, updated_at
|
||||
) VALUES (
|
||||
:operation_id, 'platform_admin_grant', 'pending_approval',
|
||||
:request_hash, :account_id, 'default', :user_id,
|
||||
'0041 migration integrity test', 'TEST-0041', '{}', '[]',
|
||||
:preview_hash, '[]', '{}', '[]', :now, :now, :now
|
||||
)
|
||||
"""
|
||||
),
|
||||
{
|
||||
"operation_id": f"migration-operation-{suffix}",
|
||||
"request_hash": (suffix[0] * 64),
|
||||
"account_id": f"migration-account-{suffix}",
|
||||
"user_id": f"migration-user-{suffix}",
|
||||
"preview_hash": (suffix[0] * 64),
|
||||
"now": now,
|
||||
},
|
||||
)
|
||||
connection.execute(
|
||||
text(
|
||||
"""
|
||||
INSERT INTO privileged_identity_operation_events (
|
||||
event_id, operation_id, sequence_no, event_type,
|
||||
status_before, status_after, trace_id, actor_account_id,
|
||||
actor_tenant_id, actor_user_id, snapshot_before,
|
||||
snapshot_after, snapshot_before_hash, snapshot_after_hash,
|
||||
created_at
|
||||
) VALUES (
|
||||
:event_id, :operation_id, 1, 'operation_submitted',
|
||||
NULL, 'pending_approval', :trace_id, NULL, 'default',
|
||||
:user_id, '{}', '{}', :before_hash, :after_hash, :now
|
||||
)
|
||||
"""
|
||||
),
|
||||
{
|
||||
"event_id": f"migration-event-{suffix}",
|
||||
"operation_id": f"migration-operation-{suffix}",
|
||||
"trace_id": f"migration-trace-{suffix}",
|
||||
"user_id": f"migration-user-{suffix}",
|
||||
"before_hash": "0" * 64,
|
||||
"after_hash": suffix[0] * 64,
|
||||
"now": now,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def test_privileged_identity_operation_migration_upgrade_and_downgrade(
|
||||
tmp_path: Path,
|
||||
monkeypatch,
|
||||
@@ -31,6 +95,7 @@ def test_privileged_identity_operation_migration_upgrade_and_downgrade(
|
||||
assert "privileged_identity_operations" in inspector.get_table_names()
|
||||
assert "privileged_identity_operation_approvals" in inspector.get_table_names()
|
||||
assert "privileged_identity_operation_events" in inspector.get_table_names()
|
||||
assert "privileged_identity_outbox_messages" in inspector.get_table_names()
|
||||
operation_columns = {
|
||||
column["name"]
|
||||
for column in inspector.get_columns("privileged_identity_operations")
|
||||
@@ -91,6 +156,7 @@ def test_privileged_identity_operation_migration_upgrade_and_downgrade(
|
||||
"ix_privileged_identity_operation_events_operation_created",
|
||||
"ix_privileged_identity_operation_events_type_created",
|
||||
"ix_privileged_identity_operation_events_trace",
|
||||
"ux_privileged_identity_operation_events_operation_event",
|
||||
} <= event_indexes
|
||||
with engine.connect() as connection:
|
||||
event_triggers = {
|
||||
@@ -102,6 +168,8 @@ def test_privileged_identity_operation_migration_upgrade_and_downgrade(
|
||||
assert {
|
||||
"trg_privileged_identity_operation_events_no_update",
|
||||
"trg_privileged_identity_operation_events_no_delete",
|
||||
"trg_privileged_identity_outbox_event_operation_insert",
|
||||
"trg_privileged_identity_outbox_event_operation_update",
|
||||
} <= event_triggers
|
||||
event_foreign_keys = inspector.get_foreign_keys(
|
||||
"privileged_identity_operation_events"
|
||||
@@ -112,18 +180,294 @@ def test_privileged_identity_operation_migration_upgrade_and_downgrade(
|
||||
if item["referred_table"] == "privileged_identity_operations"
|
||||
)
|
||||
assert operation_foreign_key.get("options", {}).get("ondelete") in {None, "NO ACTION"}
|
||||
outbox_columns = {
|
||||
column["name"]
|
||||
for column in inspector.get_columns("privileged_identity_outbox_messages")
|
||||
}
|
||||
assert outbox_columns == {
|
||||
"message_id", "operation_id", "event_id", "effect_type", "schema_version",
|
||||
"idempotency_key", "payload", "payload_hash", "status", "attempt_count",
|
||||
"max_attempts", "available_at", "lease_owner", "lease_expires_at",
|
||||
"last_error_code", "last_attempt_at", "delivered_at", "created_at",
|
||||
"updated_at",
|
||||
}
|
||||
outbox_unique_constraints = {
|
||||
item["name"]
|
||||
for item in inspector.get_unique_constraints("privileged_identity_outbox_messages")
|
||||
}
|
||||
assert {
|
||||
"ux_privileged_identity_outbox_event_effect",
|
||||
"ux_privileged_identity_outbox_idempotency_key",
|
||||
} <= outbox_unique_constraints
|
||||
outbox_indexes = {
|
||||
item["name"]
|
||||
for item in inspector.get_indexes("privileged_identity_outbox_messages")
|
||||
}
|
||||
assert {
|
||||
"ix_privileged_identity_outbox_dispatch",
|
||||
"ix_privileged_identity_outbox_operation_created",
|
||||
"ix_privileged_identity_outbox_status_lease",
|
||||
} <= outbox_indexes
|
||||
outbox_foreign_keys = inspector.get_foreign_keys(
|
||||
"privileged_identity_outbox_messages"
|
||||
)
|
||||
assert {item["referred_table"] for item in outbox_foreign_keys} == {
|
||||
"privileged_identity_operations",
|
||||
"privileged_identity_operation_events",
|
||||
}
|
||||
assert all(
|
||||
item.get("options", {}).get("ondelete") in {None, "NO ACTION"}
|
||||
for item in outbox_foreign_keys
|
||||
)
|
||||
composite_event_foreign_key = next(
|
||||
item
|
||||
for item in outbox_foreign_keys
|
||||
if item["name"] == "fk_privileged_identity_outbox_event_operation"
|
||||
)
|
||||
assert composite_event_foreign_key["constrained_columns"] == [
|
||||
"operation_id",
|
||||
"event_id",
|
||||
]
|
||||
assert composite_event_foreign_key["referred_columns"] == [
|
||||
"operation_id",
|
||||
"event_id",
|
||||
]
|
||||
outbox_check_constraints = {
|
||||
item["name"]
|
||||
for item in inspector.get_check_constraints(
|
||||
"privileged_identity_outbox_messages"
|
||||
)
|
||||
}
|
||||
assert "ck_privileged_identity_outbox_lease_state" in outbox_check_constraints
|
||||
|
||||
now = datetime.now(UTC).isoformat()
|
||||
with engine.begin() as connection:
|
||||
_insert_migration_operation_and_event(
|
||||
connection,
|
||||
suffix="pending",
|
||||
now=now,
|
||||
)
|
||||
connection.execute(
|
||||
text(
|
||||
"""
|
||||
INSERT INTO privileged_identity_outbox_messages (
|
||||
message_id, operation_id, event_id, effect_type,
|
||||
schema_version, idempotency_key, payload, payload_hash,
|
||||
status, attempt_count, max_attempts, available_at,
|
||||
lease_owner, lease_expires_at, last_error_code,
|
||||
last_attempt_at, delivered_at, created_at, updated_at
|
||||
) VALUES (
|
||||
:message_id, :operation_id, :event_id, :effect_type,
|
||||
1, :idempotency_key, '{}', :payload_hash,
|
||||
'queued', 0, 5, :now,
|
||||
NULL, NULL, NULL, NULL, NULL, :now, :now
|
||||
)
|
||||
"""
|
||||
),
|
||||
{
|
||||
"message_id": "migration-outbox-pending",
|
||||
"operation_id": "migration-operation-pending",
|
||||
"event_id": "migration-event-pending",
|
||||
"effect_type": "privileged_identity.operation_event",
|
||||
"idempotency_key": "migration-idempotency-key",
|
||||
"payload_hash": "0" * 64,
|
||||
"now": now,
|
||||
},
|
||||
)
|
||||
with pytest.raises(RuntimeError, match="undelivered messages remain"):
|
||||
command.downgrade(config, "20260802_0039")
|
||||
with engine.begin() as connection:
|
||||
connection.execute(
|
||||
text(
|
||||
"DELETE FROM privileged_identity_outbox_messages "
|
||||
"WHERE message_id = 'migration-outbox-pending'"
|
||||
)
|
||||
)
|
||||
|
||||
command.downgrade(config, "20260802_0039")
|
||||
downgraded_to_0039 = inspect(engine)
|
||||
assert "privileged_identity_outbox_messages" not in downgraded_to_0039.get_table_names()
|
||||
assert "privileged_identity_operation_events" in downgraded_to_0039.get_table_names()
|
||||
command.upgrade(config, "20260802_0040")
|
||||
assert "privileged_identity_outbox_messages" in inspect(engine).get_table_names()
|
||||
|
||||
command.downgrade(config, "20260802_0036")
|
||||
downgraded_tables = inspect(engine).get_table_names()
|
||||
assert "privileged_identity_operations" not in downgraded_tables
|
||||
assert "privileged_identity_operation_approvals" not in downgraded_tables
|
||||
assert "privileged_identity_operation_events" not in downgraded_tables
|
||||
assert "privileged_identity_outbox_messages" not in downgraded_tables
|
||||
|
||||
command.upgrade(config, "head")
|
||||
reupgraded_tables = inspect(engine).get_table_names()
|
||||
assert "privileged_identity_operations" in reupgraded_tables
|
||||
assert "privileged_identity_operation_approvals" in reupgraded_tables
|
||||
assert "privileged_identity_operation_events" in reupgraded_tables
|
||||
assert "privileged_identity_outbox_messages" in reupgraded_tables
|
||||
engine.dispose()
|
||||
finally:
|
||||
get_settings.cache_clear()
|
||||
|
||||
|
||||
def test_0041_rejects_invalid_existing_rows_and_enforces_outbox_integrity(
|
||||
tmp_path: Path,
|
||||
monkeypatch,
|
||||
) -> None:
|
||||
database_path = tmp_path / "privileged-identity-outbox-integrity.sqlite3"
|
||||
database_url = f"sqlite+pysqlite:///{database_path.as_posix()}"
|
||||
monkeypatch.setenv("CONTRACT_API_DATABASE_URL", database_url)
|
||||
get_settings.cache_clear()
|
||||
config = Config(str(SERVICE_ROOT / "alembic.ini"))
|
||||
|
||||
try:
|
||||
command.upgrade(config, "20260802_0040")
|
||||
engine = create_engine(database_url)
|
||||
now = datetime.now(UTC).isoformat()
|
||||
with engine.begin() as connection:
|
||||
for suffix in ("a", "b"):
|
||||
_insert_migration_operation_and_event(
|
||||
connection,
|
||||
suffix=suffix,
|
||||
now=now,
|
||||
)
|
||||
connection.execute(
|
||||
text(
|
||||
"""
|
||||
INSERT INTO privileged_identity_outbox_messages (
|
||||
message_id, operation_id, event_id, effect_type,
|
||||
schema_version, idempotency_key, payload, payload_hash,
|
||||
status, attempt_count, max_attempts, available_at,
|
||||
lease_owner, lease_expires_at, last_error_code,
|
||||
last_attempt_at, delivered_at, created_at, updated_at
|
||||
) VALUES (
|
||||
'migration-cross-operation', 'migration-operation-a',
|
||||
'migration-event-b', 'privileged_identity.operation_event',
|
||||
1, 'migration-cross-operation-key', '{}', :payload_hash,
|
||||
'queued', 0, 5, :now, NULL, NULL, NULL, NULL, NULL,
|
||||
:now, :now
|
||||
)
|
||||
"""
|
||||
),
|
||||
{"payload_hash": "0" * 64, "now": now},
|
||||
)
|
||||
|
||||
with pytest.raises(RuntimeError, match="event/operation mismatches exist"):
|
||||
command.upgrade(config, "head")
|
||||
|
||||
with engine.begin() as connection:
|
||||
connection.execute(
|
||||
text(
|
||||
"DELETE FROM privileged_identity_outbox_messages "
|
||||
"WHERE message_id = 'migration-cross-operation'"
|
||||
)
|
||||
)
|
||||
connection.execute(
|
||||
text(
|
||||
"""
|
||||
INSERT INTO privileged_identity_outbox_messages (
|
||||
message_id, operation_id, event_id, effect_type,
|
||||
schema_version, idempotency_key, payload, payload_hash,
|
||||
status, attempt_count, max_attempts, available_at,
|
||||
lease_owner, lease_expires_at, last_error_code,
|
||||
last_attempt_at, delivered_at, created_at, updated_at
|
||||
) VALUES (
|
||||
'migration-invalid-lease', 'migration-operation-a',
|
||||
'migration-event-a', 'privileged_identity.operation_event',
|
||||
1, 'migration-invalid-lease-key', '{}', :payload_hash,
|
||||
'processing', 1, 5, :now, NULL, NULL, NULL, :now, NULL,
|
||||
:now, :now
|
||||
)
|
||||
"""
|
||||
),
|
||||
{"payload_hash": "0" * 64, "now": now},
|
||||
)
|
||||
|
||||
with pytest.raises(RuntimeError, match="invalid lease state rows exist"):
|
||||
command.upgrade(config, "head")
|
||||
|
||||
with engine.begin() as connection:
|
||||
connection.execute(
|
||||
text(
|
||||
"DELETE FROM privileged_identity_outbox_messages "
|
||||
"WHERE message_id = 'migration-invalid-lease'"
|
||||
)
|
||||
)
|
||||
command.upgrade(config, "head")
|
||||
|
||||
with engine.connect() as connection:
|
||||
connection.exec_driver_sql("PRAGMA foreign_keys=OFF")
|
||||
assert connection.exec_driver_sql("PRAGMA foreign_keys").scalar_one() == 0
|
||||
connection.commit()
|
||||
connection.execute(
|
||||
text(
|
||||
"""
|
||||
INSERT INTO privileged_identity_outbox_messages (
|
||||
message_id, operation_id, event_id, effect_type,
|
||||
schema_version, idempotency_key, payload, payload_hash,
|
||||
status, attempt_count, max_attempts, available_at,
|
||||
lease_owner, lease_expires_at, last_error_code,
|
||||
last_attempt_at, delivered_at, created_at, updated_at
|
||||
) VALUES (
|
||||
'migration-valid-outbox', 'migration-operation-a',
|
||||
'migration-event-a', 'privileged_identity.operation_event',
|
||||
1, 'migration-valid-key', '{}', :payload_hash,
|
||||
'queued', 0, 5, :now, NULL, NULL, NULL, NULL, NULL,
|
||||
:now, :now
|
||||
)
|
||||
"""
|
||||
),
|
||||
{"payload_hash": "0" * 64, "now": now},
|
||||
)
|
||||
connection.commit()
|
||||
|
||||
with pytest.raises(DBAPIError):
|
||||
connection.execute(
|
||||
text(
|
||||
"UPDATE privileged_identity_outbox_messages "
|
||||
"SET event_id = 'migration-event-b' "
|
||||
"WHERE message_id = 'migration-valid-outbox'"
|
||||
)
|
||||
)
|
||||
connection.commit()
|
||||
connection.rollback()
|
||||
|
||||
invalid_lease_assignments = (
|
||||
"status = 'processing', lease_owner = NULL, lease_expires_at = NULL",
|
||||
"status = 'processing', lease_owner = 'worker', lease_expires_at = NULL",
|
||||
"status = 'processing', lease_owner = NULL, lease_expires_at = :now",
|
||||
"status = 'queued', lease_owner = 'worker', lease_expires_at = NULL",
|
||||
"status = 'queued', lease_owner = NULL, lease_expires_at = :now",
|
||||
"status = 'queued', lease_owner = 'worker', lease_expires_at = :now",
|
||||
)
|
||||
for assignments in invalid_lease_assignments:
|
||||
with pytest.raises(DBAPIError):
|
||||
connection.execute(
|
||||
text(
|
||||
"UPDATE privileged_identity_outbox_messages "
|
||||
f"SET {assignments} "
|
||||
"WHERE message_id = 'migration-valid-outbox'"
|
||||
),
|
||||
{"now": now},
|
||||
)
|
||||
connection.commit()
|
||||
connection.rollback()
|
||||
|
||||
valid_lease_assignments = (
|
||||
"status = 'processing', lease_owner = 'worker', lease_expires_at = :now",
|
||||
"status = 'retry_scheduled', lease_owner = NULL, lease_expires_at = NULL",
|
||||
"status = 'dead_letter', lease_owner = NULL, lease_expires_at = NULL",
|
||||
"status = 'delivered', lease_owner = NULL, lease_expires_at = NULL",
|
||||
)
|
||||
for assignments in valid_lease_assignments:
|
||||
connection.execute(
|
||||
text(
|
||||
"UPDATE privileged_identity_outbox_messages "
|
||||
f"SET {assignments} "
|
||||
"WHERE message_id = 'migration-valid-outbox'"
|
||||
),
|
||||
{"now": now},
|
||||
)
|
||||
connection.commit()
|
||||
engine.dispose()
|
||||
finally:
|
||||
get_settings.cache_clear()
|
||||
|
||||
+890
-2
@@ -2,9 +2,10 @@ import json
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from threading import Barrier
|
||||
from threading import Barrier, Event
|
||||
|
||||
import pytest
|
||||
from fastapi import HTTPException
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import select, text
|
||||
from sqlalchemy.exc import DBAPIError
|
||||
@@ -17,6 +18,7 @@ from yuqei_contract_api.persistence.models import (
|
||||
PrivilegedIdentityOperationApprovalModel,
|
||||
PrivilegedIdentityOperationEventModel,
|
||||
PrivilegedIdentityOperationModel,
|
||||
PrivilegedIdentityOutboxMessageModel,
|
||||
TenantUserModel,
|
||||
WorkspaceMembershipModel,
|
||||
)
|
||||
@@ -24,6 +26,10 @@ from yuqei_contract_api.privileged_identity import (
|
||||
PrivilegedIdentityOperationError,
|
||||
privileged_payload_hash,
|
||||
)
|
||||
from yuqei_contract_api.persistence.privileged_identity_outbox import (
|
||||
PrivilegedIdentityOutboxRepository,
|
||||
PrivilegedIdentityOutboxWorker,
|
||||
)
|
||||
|
||||
|
||||
AUTH_VALUE = "x" * 48
|
||||
@@ -155,6 +161,71 @@ def submit_from_preview(
|
||||
return response.json()
|
||||
|
||||
|
||||
def governance_storage_snapshot(repository) -> tuple[tuple, tuple, tuple]:
|
||||
with repository._session_factory() as session:
|
||||
operations = session.execute(
|
||||
select(PrivilegedIdentityOperationModel).order_by(
|
||||
PrivilegedIdentityOperationModel.operation_id
|
||||
)
|
||||
).scalars().all()
|
||||
events = session.execute(
|
||||
select(PrivilegedIdentityOperationEventModel).order_by(
|
||||
PrivilegedIdentityOperationEventModel.event_id
|
||||
)
|
||||
).scalars().all()
|
||||
messages = session.execute(
|
||||
select(PrivilegedIdentityOutboxMessageModel).order_by(
|
||||
PrivilegedIdentityOutboxMessageModel.message_id
|
||||
)
|
||||
).scalars().all()
|
||||
return (
|
||||
tuple(
|
||||
(
|
||||
item.operation_id,
|
||||
item.status,
|
||||
item.request_payload_hash,
|
||||
item.preview_hash,
|
||||
item.updated_at,
|
||||
)
|
||||
for item in operations
|
||||
),
|
||||
tuple(
|
||||
(
|
||||
item.event_id,
|
||||
item.operation_id,
|
||||
item.event_type,
|
||||
item.status_before,
|
||||
item.status_after,
|
||||
item.snapshot_before_hash,
|
||||
item.snapshot_after_hash,
|
||||
item.created_at,
|
||||
)
|
||||
for item in events
|
||||
),
|
||||
tuple(
|
||||
(
|
||||
item.message_id,
|
||||
item.operation_id,
|
||||
item.event_id,
|
||||
item.status,
|
||||
item.attempt_count,
|
||||
item.max_attempts,
|
||||
item.idempotency_key,
|
||||
json.dumps(item.payload, sort_keys=True),
|
||||
item.payload_hash,
|
||||
item.available_at,
|
||||
item.lease_owner,
|
||||
item.lease_expires_at,
|
||||
item.last_error_code,
|
||||
item.last_attempt_at,
|
||||
item.delivered_at,
|
||||
item.updated_at,
|
||||
)
|
||||
for item in messages
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def test_preview_is_platform_only_read_only_and_minimally_disclosed(tmp_path: Path) -> None:
|
||||
app = create_app(make_settings(tmp_path))
|
||||
client = TestClient(app)
|
||||
@@ -190,11 +261,17 @@ def test_preview_is_platform_only_read_only_and_minimally_disclosed(tmp_path: Pa
|
||||
"/api/v1/identity/privileged-operations/operation-not-visible/events",
|
||||
headers=identity_headers("user-governed-target"),
|
||||
)
|
||||
delivery_forbidden = client.get(
|
||||
"/api/v1/identity/privileged-operations/operation-not-visible/delivery-status",
|
||||
headers=identity_headers("user-governed-target"),
|
||||
)
|
||||
|
||||
assert forbidden.status_code == 403
|
||||
assert forbidden.json()["detail"]["code"] == "platform_admin_required"
|
||||
assert events_forbidden.status_code == 403
|
||||
assert events_forbidden.json()["detail"]["code"] == "platform_admin_required"
|
||||
assert delivery_forbidden.status_code == 403
|
||||
assert delivery_forbidden.json()["detail"]["code"] == "platform_admin_required"
|
||||
assert preview["operation_type"] == "platform_admin_grant"
|
||||
assert preview["impact_summary"]["platform_permissions_changed"] == 1
|
||||
assert preview["blocking_issues"] == []
|
||||
@@ -250,6 +327,10 @@ def test_preview_returns_stable_400_404_and_422_errors(tmp_path: Path) -> None:
|
||||
"/api/v1/identity/privileged-operations/operation-missing/events",
|
||||
headers=identity_headers(),
|
||||
)
|
||||
missing_delivery = client.get(
|
||||
"/api/v1/identity/privileged-operations/operation-missing/delivery-status",
|
||||
headers=identity_headers(),
|
||||
)
|
||||
|
||||
assert invalid_json_value.status_code == 400
|
||||
assert invalid_json_value.json()["detail"]["code"] == "invalid_target_refs"
|
||||
@@ -262,6 +343,8 @@ def test_preview_returns_stable_400_404_and_422_errors(tmp_path: Path) -> None:
|
||||
assert "must-not-be-stored" not in unexpected_sensitive_ref.text
|
||||
assert missing_events.status_code == 404
|
||||
assert missing_events.json()["detail"]["code"] == "operation_not_found"
|
||||
assert missing_delivery.status_code == 404
|
||||
assert missing_delivery.json()["detail"]["code"] == "operation_not_found"
|
||||
|
||||
|
||||
def test_list_and_detail_map_governance_repository_errors(
|
||||
@@ -398,6 +481,55 @@ def test_submission_is_persistent_idempotent_and_rejects_stale_preview(tmp_path:
|
||||
)
|
||||
).scalars().all()
|
||||
assert len(stored_events) == 1
|
||||
stored_outbox = session.execute(
|
||||
select(PrivilegedIdentityOutboxMessageModel).where(
|
||||
PrivilegedIdentityOutboxMessageModel.operation_id
|
||||
== "operation-idempotent-001"
|
||||
)
|
||||
).scalars().all()
|
||||
assert len(stored_outbox) == 1
|
||||
assert stored_outbox[0].event_id == stored_events[0].event_id
|
||||
assert stored_outbox[0].payload_hash == privileged_payload_hash(
|
||||
stored_outbox[0].payload
|
||||
)
|
||||
assert set(stored_outbox[0].payload) == {
|
||||
"event_id",
|
||||
"operation_id",
|
||||
"sequence_no",
|
||||
"event_type",
|
||||
"operation_type",
|
||||
"status_before",
|
||||
"status_after",
|
||||
"approval_count",
|
||||
"target_reference_count",
|
||||
"target_reference_fingerprints",
|
||||
"evidence_reference_count",
|
||||
"warning_count",
|
||||
"trace_id",
|
||||
"snapshot_before_hash",
|
||||
"snapshot_after_hash",
|
||||
"occurred_at",
|
||||
}
|
||||
serialized_outbox = json.dumps(stored_outbox[0].payload).lower()
|
||||
assert "user-governed-target" not in serialized_outbox
|
||||
for forbidden_term in (
|
||||
"email", "name", "token", "credential", "secret", "password",
|
||||
"governed-target@example.com", "sup-2026-0802-001",
|
||||
):
|
||||
assert forbidden_term not in serialized_outbox
|
||||
|
||||
delivery_status = client.get(
|
||||
"/api/v1/identity/privileged-operations/operation-idempotent-001/delivery-status",
|
||||
headers=identity_headers(),
|
||||
)
|
||||
assert delivery_status.status_code == 200
|
||||
assert delivery_status.json()["total"] == 1
|
||||
delivery_item = delivery_status.json()["items"][0]
|
||||
assert delivery_item["status"] == "queued"
|
||||
assert delivery_item["event_type"] == "operation_submitted"
|
||||
assert "payload" not in delivery_item
|
||||
assert "lease_owner" not in delivery_item
|
||||
assert delivery_item["last_error_code"] is None
|
||||
|
||||
with pytest.raises(DBAPIError):
|
||||
with repository._session_factory.begin() as session:
|
||||
@@ -418,6 +550,31 @@ def test_submission_is_persistent_idempotent_and_rejects_stale_preview(tmp_path:
|
||||
{"event_id": submitted_event["event_id"]},
|
||||
)
|
||||
|
||||
second_preview = grant_preview(client)
|
||||
submit_from_preview(
|
||||
client,
|
||||
second_preview,
|
||||
operation_id="operation-idempotent-002",
|
||||
)
|
||||
with repository._session_factory() as session:
|
||||
second_event_id = session.scalar(
|
||||
select(PrivilegedIdentityOperationEventModel.event_id).where(
|
||||
PrivilegedIdentityOperationEventModel.operation_id
|
||||
== "operation-idempotent-002"
|
||||
)
|
||||
)
|
||||
assert second_event_id is not None
|
||||
with pytest.raises(DBAPIError):
|
||||
with repository._session_factory.begin() as session:
|
||||
session.execute(
|
||||
text(
|
||||
"UPDATE privileged_identity_outbox_messages "
|
||||
"SET event_id = :event_id "
|
||||
"WHERE operation_id = 'operation-idempotent-001'"
|
||||
),
|
||||
{"event_id": second_event_id},
|
||||
)
|
||||
|
||||
stale_preview = grant_preview(client)
|
||||
with repository._session_factory.begin() as session:
|
||||
target = session.get(TenantUserModel, "user-governed-target")
|
||||
@@ -498,6 +655,10 @@ def test_two_distinct_current_admins_are_required_and_requester_cannot_approve(t
|
||||
"/api/v1/identity/privileged-operations/operation-two-person-001/events",
|
||||
headers=identity_headers(),
|
||||
)
|
||||
delivery = client.get(
|
||||
"/api/v1/identity/privileged-operations/operation-two-person-001/delivery-status",
|
||||
headers=identity_headers(),
|
||||
)
|
||||
|
||||
assert self_approval.status_code == 403
|
||||
assert self_approval.json()["detail"]["code"] == "requester_cannot_approve"
|
||||
@@ -522,6 +683,11 @@ def test_two_distinct_current_admins_are_required_and_requester_cannot_approve(t
|
||||
assert events.json()["items"][2]["status_after"] == "approved"
|
||||
assert events.json()["items"][1]["trace_id"] == "trace-approval-one-001"
|
||||
assert events.json()["items"][2]["trace_id"] == "trace-approval-two-001"
|
||||
assert delivery.status_code == 200
|
||||
assert delivery.json()["total"] == 3
|
||||
assert [item["event_id"] for item in delivery.json()["items"]] == [
|
||||
item["event_id"] for item in events.json()["items"]
|
||||
]
|
||||
with repository._session_factory() as session:
|
||||
target = session.get(TenantUserModel, "user-governed-target")
|
||||
assert target is not None
|
||||
@@ -585,6 +751,13 @@ def test_same_approver_concurrent_decisions_are_idempotent(tmp_path: Path) -> No
|
||||
).scalars().all()
|
||||
assert len(events) == 2
|
||||
assert sorted(event.sequence_no for event in events) == [1, 2]
|
||||
outbox_messages = session.execute(
|
||||
select(PrivilegedIdentityOutboxMessageModel).where(
|
||||
PrivilegedIdentityOutboxMessageModel.operation_id
|
||||
== "operation-concurrent-001"
|
||||
)
|
||||
).scalars().all()
|
||||
assert len(outbox_messages) == 2
|
||||
|
||||
|
||||
def test_rejection_is_terminal_and_expired_operation_returns_410(tmp_path: Path) -> None:
|
||||
@@ -688,6 +861,20 @@ def test_rejection_is_terminal_and_expired_operation_returns_410(tmp_path: Path)
|
||||
)
|
||||
).scalars().all()
|
||||
assert len(stored_events) == 2
|
||||
expired_outbox = session.execute(
|
||||
select(PrivilegedIdentityOutboxMessageModel).where(
|
||||
PrivilegedIdentityOutboxMessageModel.operation_id
|
||||
== "operation-expired-001"
|
||||
)
|
||||
).scalars().all()
|
||||
rejected_outbox = session.execute(
|
||||
select(PrivilegedIdentityOutboxMessageModel).where(
|
||||
PrivilegedIdentityOutboxMessageModel.operation_id
|
||||
== "operation-rejected-001"
|
||||
)
|
||||
).scalars().all()
|
||||
assert len(expired_outbox) == 2
|
||||
assert len(rejected_outbox) == 2
|
||||
|
||||
|
||||
def test_privileged_operation_routes_are_s3_sensitive() -> None:
|
||||
@@ -699,6 +886,7 @@ def test_privileged_operation_routes_are_s3_sensitive() -> None:
|
||||
("GET", "/api/v1/identity/privileged-operations"),
|
||||
("GET", "/api/v1/identity/privileged-operations/op-001"),
|
||||
("GET", "/api/v1/identity/privileged-operations/op-001/events"),
|
||||
("GET", "/api/v1/identity/privileged-operations/op-001/delivery-status"),
|
||||
):
|
||||
policy = match_policy(method, path, policies)
|
||||
assert policy is not None
|
||||
@@ -707,9 +895,20 @@ def test_privileged_operation_routes_are_s3_sensitive() -> None:
|
||||
|
||||
|
||||
def test_no_execution_or_destructive_identity_route_is_registered(tmp_path: Path) -> None:
|
||||
openapi_paths = create_app(make_settings(tmp_path)).openapi()["paths"]
|
||||
for path in (
|
||||
"/api/v1/identity/privileged-outbox/overview",
|
||||
"/api/v1/identity/privileged-outbox/dead-letters",
|
||||
):
|
||||
assert path in openapi_paths
|
||||
assert "get" in openapi_paths[path]
|
||||
assert not any(
|
||||
method in openapi_paths[path]
|
||||
for method in ("post", "put", "patch", "delete")
|
||||
)
|
||||
paths = {
|
||||
path
|
||||
for path in create_app(make_settings(tmp_path)).openapi()["paths"]
|
||||
for path in openapi_paths
|
||||
if "/identity/" in path
|
||||
}
|
||||
forbidden_suffixes = (
|
||||
@@ -720,5 +919,694 @@ def test_no_execution_or_destructive_identity_route_is_registered(tmp_path: Path
|
||||
"/recover",
|
||||
"/grant",
|
||||
"/revoke",
|
||||
"/support-access",
|
||||
)
|
||||
assert not any(path.endswith(forbidden_suffixes) for path in paths)
|
||||
assert not any(path.endswith("/retry") for path in paths)
|
||||
destructive_write_routes = [
|
||||
(method.upper(), path)
|
||||
for path, operations in openapi_paths.items()
|
||||
if "/identity/" in path
|
||||
for method in operations
|
||||
if method.upper() in {"POST", "PUT", "PATCH", "DELETE"}
|
||||
and any(segment in path for segment in (*forbidden_suffixes, "/retry"))
|
||||
]
|
||||
assert destructive_write_routes == []
|
||||
|
||||
|
||||
def test_outbox_failure_rolls_back_operation_and_event(
|
||||
tmp_path: Path,
|
||||
monkeypatch,
|
||||
) -> None:
|
||||
app = create_app(make_settings(tmp_path))
|
||||
client = TestClient(app)
|
||||
identity_repository = app.state.contract_repository.identity
|
||||
add_identity(
|
||||
identity_repository,
|
||||
account_id="account-governed-target",
|
||||
user_id="user-governed-target",
|
||||
email="governed-target@example.com",
|
||||
platform_admin=False,
|
||||
)
|
||||
preview = grant_preview(client)
|
||||
|
||||
def fail_outbox(*_args, **_kwargs) -> None:
|
||||
raise RuntimeError("injected outbox failure")
|
||||
|
||||
monkeypatch.setattr(
|
||||
identity_repository.privileged_operations,
|
||||
"_append_outbox_message",
|
||||
fail_outbox,
|
||||
)
|
||||
with pytest.raises(HTTPException) as injected_error:
|
||||
submit_from_preview(
|
||||
client,
|
||||
preview,
|
||||
operation_id="operation-outbox-rollback-001",
|
||||
)
|
||||
assert injected_error.value.status_code == 500
|
||||
assert "injected outbox failure" in str(injected_error.value.detail)
|
||||
|
||||
with identity_repository._session_factory() as session:
|
||||
assert session.get(
|
||||
PrivilegedIdentityOperationModel,
|
||||
"operation-outbox-rollback-001",
|
||||
) is None
|
||||
assert session.execute(
|
||||
select(PrivilegedIdentityOperationEventModel).where(
|
||||
PrivilegedIdentityOperationEventModel.operation_id
|
||||
== "operation-outbox-rollback-001"
|
||||
)
|
||||
).scalars().all() == []
|
||||
assert session.execute(
|
||||
select(PrivilegedIdentityOutboxMessageModel).where(
|
||||
PrivilegedIdentityOutboxMessageModel.operation_id
|
||||
== "operation-outbox-rollback-001"
|
||||
)
|
||||
).scalars().all() == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("decision", ["approve", "reject"])
|
||||
def test_outbox_failure_rolls_back_approval_or_rejection(
|
||||
tmp_path: Path,
|
||||
monkeypatch,
|
||||
decision: str,
|
||||
) -> None:
|
||||
app = create_app(make_settings(tmp_path))
|
||||
client = TestClient(app)
|
||||
identity_repository = app.state.contract_repository.identity
|
||||
add_identity(
|
||||
identity_repository,
|
||||
account_id="account-governed-target",
|
||||
user_id="user-governed-target",
|
||||
email="governed-target@example.com",
|
||||
platform_admin=False,
|
||||
)
|
||||
add_identity(
|
||||
identity_repository,
|
||||
account_id="account-outbox-approver",
|
||||
user_id="user-outbox-approver",
|
||||
email="outbox-approver@example.com",
|
||||
platform_admin=True,
|
||||
)
|
||||
preview = grant_preview(client)
|
||||
operation = submit_from_preview(
|
||||
client,
|
||||
preview,
|
||||
operation_id=f"operation-outbox-{decision}-rollback-001",
|
||||
)
|
||||
|
||||
def fail_outbox(*_args, **_kwargs) -> None:
|
||||
raise RuntimeError("injected decision outbox failure")
|
||||
|
||||
monkeypatch.setattr(
|
||||
identity_repository.privileged_operations,
|
||||
"_append_outbox_message",
|
||||
fail_outbox,
|
||||
)
|
||||
with pytest.raises(HTTPException) as injected_error:
|
||||
client.post(
|
||||
f"/api/v1/identity/privileged-operations/{operation['operation_id']}/approvals",
|
||||
headers=identity_headers("user-outbox-approver"),
|
||||
json={
|
||||
"decision": decision,
|
||||
"comment": "Failure injection must roll back the complete decision",
|
||||
"preview_hash": operation["preview_hash"],
|
||||
},
|
||||
)
|
||||
assert injected_error.value.status_code == 500
|
||||
|
||||
with identity_repository._session_factory() as session:
|
||||
stored = session.get(PrivilegedIdentityOperationModel, operation["operation_id"])
|
||||
assert stored is not None
|
||||
assert stored.status == "pending_approval"
|
||||
approvals = session.execute(
|
||||
select(PrivilegedIdentityOperationApprovalModel).where(
|
||||
PrivilegedIdentityOperationApprovalModel.operation_id
|
||||
== operation["operation_id"]
|
||||
)
|
||||
).scalars().all()
|
||||
events = session.execute(
|
||||
select(PrivilegedIdentityOperationEventModel).where(
|
||||
PrivilegedIdentityOperationEventModel.operation_id
|
||||
== operation["operation_id"]
|
||||
)
|
||||
).scalars().all()
|
||||
outbox_messages = session.execute(
|
||||
select(PrivilegedIdentityOutboxMessageModel).where(
|
||||
PrivilegedIdentityOutboxMessageModel.operation_id
|
||||
== operation["operation_id"]
|
||||
)
|
||||
).scalars().all()
|
||||
assert approvals == []
|
||||
assert len(events) == 1
|
||||
assert len(outbox_messages) == 1
|
||||
|
||||
|
||||
def test_delivery_status_is_read_only_and_expiry_outbox_failure_rolls_back(
|
||||
tmp_path: Path,
|
||||
monkeypatch,
|
||||
) -> None:
|
||||
app = create_app(make_settings(tmp_path))
|
||||
client = TestClient(app)
|
||||
identity_repository = app.state.contract_repository.identity
|
||||
add_identity(
|
||||
identity_repository,
|
||||
account_id="account-governed-target",
|
||||
user_id="user-governed-target",
|
||||
email="governed-target@example.com",
|
||||
platform_admin=False,
|
||||
)
|
||||
preview = grant_preview(client)
|
||||
operation = submit_from_preview(
|
||||
client,
|
||||
preview,
|
||||
operation_id="operation-outbox-expiry-rollback-001",
|
||||
)
|
||||
with identity_repository._session_factory.begin() as session:
|
||||
stored = session.get(PrivilegedIdentityOperationModel, operation["operation_id"])
|
||||
assert stored is not None
|
||||
stored.expires_at = datetime.now(UTC) - timedelta(seconds=1)
|
||||
|
||||
delivery_status = client.get(
|
||||
f"/api/v1/identity/privileged-operations/{operation['operation_id']}/delivery-status",
|
||||
headers=identity_headers(),
|
||||
)
|
||||
assert delivery_status.status_code == 200
|
||||
assert delivery_status.json()["total"] == 1
|
||||
with identity_repository._session_factory() as session:
|
||||
stored = session.get(PrivilegedIdentityOperationModel, operation["operation_id"])
|
||||
assert stored is not None
|
||||
assert stored.status == "pending_approval"
|
||||
|
||||
def fail_outbox(*_args, **_kwargs) -> None:
|
||||
raise RuntimeError("injected expiry outbox failure")
|
||||
|
||||
monkeypatch.setattr(
|
||||
identity_repository.privileged_operations,
|
||||
"_append_outbox_message",
|
||||
fail_outbox,
|
||||
)
|
||||
with pytest.raises(HTTPException) as injected_error:
|
||||
client.get(
|
||||
f"/api/v1/identity/privileged-operations/{operation['operation_id']}",
|
||||
headers=identity_headers(),
|
||||
)
|
||||
assert injected_error.value.status_code == 500
|
||||
|
||||
with identity_repository._session_factory() as session:
|
||||
stored = session.get(PrivilegedIdentityOperationModel, operation["operation_id"])
|
||||
assert stored is not None
|
||||
assert stored.status == "pending_approval"
|
||||
events = session.execute(
|
||||
select(PrivilegedIdentityOperationEventModel).where(
|
||||
PrivilegedIdentityOperationEventModel.operation_id
|
||||
== operation["operation_id"]
|
||||
)
|
||||
).scalars().all()
|
||||
outbox_messages = session.execute(
|
||||
select(PrivilegedIdentityOutboxMessageModel).where(
|
||||
PrivilegedIdentityOutboxMessageModel.operation_id
|
||||
== operation["operation_id"]
|
||||
)
|
||||
).scalars().all()
|
||||
assert len(events) == 1
|
||||
assert len(outbox_messages) == 1
|
||||
|
||||
|
||||
def test_privileged_outbox_observability_is_platform_only_and_empty_is_healthy(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
app = create_app(make_settings(tmp_path))
|
||||
client = TestClient(app)
|
||||
identity_repository = app.state.contract_repository.identity
|
||||
add_identity(
|
||||
identity_repository,
|
||||
account_id="account-tenant-admin",
|
||||
user_id="user-tenant-admin",
|
||||
email="tenant-admin@example.com",
|
||||
platform_admin=False,
|
||||
)
|
||||
with identity_repository._session_factory.begin() as session:
|
||||
tenant_admin = session.get(TenantUserModel, "user-tenant-admin")
|
||||
assert tenant_admin is not None
|
||||
tenant_admin.role = "admin"
|
||||
|
||||
for path in (
|
||||
"/api/v1/identity/privileged-outbox/overview",
|
||||
"/api/v1/identity/privileged-outbox/dead-letters",
|
||||
):
|
||||
forbidden = client.get(path, headers=identity_headers("user-tenant-admin"))
|
||||
assert forbidden.status_code == 403
|
||||
assert forbidden.json()["detail"]["code"] == "platform_admin_required"
|
||||
|
||||
overview = client.get(
|
||||
"/api/v1/identity/privileged-outbox/overview",
|
||||
headers=identity_headers(),
|
||||
)
|
||||
dead_letters = client.get(
|
||||
"/api/v1/identity/privileged-outbox/dead-letters",
|
||||
headers=identity_headers(),
|
||||
)
|
||||
assert overview.status_code == 200
|
||||
assert overview.json() == {
|
||||
"health_status": "healthy",
|
||||
"generated_at": overview.json()["generated_at"],
|
||||
"stale_after_seconds": 900,
|
||||
"status_counts": {
|
||||
"queued": 0,
|
||||
"processing": 0,
|
||||
"retry_scheduled": 0,
|
||||
"delivered": 0,
|
||||
"dead_letter": 0,
|
||||
},
|
||||
"active_count": 0,
|
||||
"oldest_active_age_seconds": None,
|
||||
"stale_active_count": 0,
|
||||
"expired_processing_lease_count": 0,
|
||||
"near_attempt_limit_count": 0,
|
||||
"dead_letter_count": 0,
|
||||
"dead_letter_error_codes": [],
|
||||
}
|
||||
assert dead_letters.status_code == 200
|
||||
assert dead_letters.json() == {
|
||||
"items": [],
|
||||
"total": 0,
|
||||
"page": 1,
|
||||
"page_size": 20,
|
||||
"total_pages": 0,
|
||||
}
|
||||
assert client.get(
|
||||
"/api/v1/identity/privileged-outbox/overview?stale_after_seconds=59",
|
||||
headers=identity_headers(),
|
||||
).status_code == 422
|
||||
assert client.get(
|
||||
"/api/v1/identity/privileged-outbox/overview?stale_after_seconds=86401",
|
||||
headers=identity_headers(),
|
||||
).status_code == 422
|
||||
assert client.get(
|
||||
"/api/v1/identity/privileged-outbox/overview?stale_after_seconds=60",
|
||||
headers=identity_headers(),
|
||||
).status_code == 200
|
||||
assert client.get(
|
||||
"/api/v1/identity/privileged-outbox/overview?stale_after_seconds=86400",
|
||||
headers=identity_headers(),
|
||||
).status_code == 200
|
||||
|
||||
|
||||
def test_privileged_outbox_overview_dead_letters_and_gets_are_strictly_read_only(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
app = create_app(make_settings(tmp_path))
|
||||
client = TestClient(app)
|
||||
identity_repository = app.state.contract_repository.identity
|
||||
add_identity(
|
||||
identity_repository,
|
||||
account_id="account-observability-target",
|
||||
user_id="user-observability-target",
|
||||
email="observability-target@example.com",
|
||||
platform_admin=False,
|
||||
)
|
||||
preview = grant_preview(client, user_id="user-observability-target")
|
||||
operation_ids = [
|
||||
"operation-observe-queued",
|
||||
"operation-observe-processing",
|
||||
"operation-observe-retry",
|
||||
"operation-observe-delivered",
|
||||
"operation-observe-dead-a1",
|
||||
"operation-observe-dead-a2",
|
||||
"operation-observe-dead-b",
|
||||
]
|
||||
for operation_id in operation_ids:
|
||||
submit_from_preview(
|
||||
client,
|
||||
preview,
|
||||
operation_id=operation_id,
|
||||
trace_id=f"trace-{operation_id}",
|
||||
)
|
||||
|
||||
now = datetime.now(UTC)
|
||||
with identity_repository._session_factory.begin() as session:
|
||||
messages = {
|
||||
item.operation_id: item
|
||||
for item in session.execute(
|
||||
select(PrivilegedIdentityOutboxMessageModel).where(
|
||||
PrivilegedIdentityOutboxMessageModel.operation_id.in_(operation_ids)
|
||||
)
|
||||
).scalars().all()
|
||||
}
|
||||
processing = messages["operation-observe-processing"]
|
||||
processing.status = "processing"
|
||||
processing.attempt_count = 1
|
||||
processing.lease_owner = "secret-lease-owner"
|
||||
processing.lease_expires_at = now - timedelta(seconds=1)
|
||||
processing.last_attempt_at = now - timedelta(seconds=30)
|
||||
|
||||
retry = messages["operation-observe-retry"]
|
||||
retry.status = "retry_scheduled"
|
||||
retry.attempt_count = retry.max_attempts - 1
|
||||
retry.created_at = now - timedelta(seconds=901)
|
||||
retry.updated_at = now - timedelta(seconds=30)
|
||||
|
||||
delivered = messages["operation-observe-delivered"]
|
||||
delivered.status = "delivered"
|
||||
delivered.attempt_count = 1
|
||||
delivered.delivered_at = now - timedelta(seconds=10)
|
||||
|
||||
dead_letter_specs = (
|
||||
("operation-observe-dead-a1", "transport_error", now - timedelta(minutes=1)),
|
||||
("operation-observe-dead-a2", "transport_error", now - timedelta(minutes=1)),
|
||||
("operation-observe-dead-b", None, now - timedelta(minutes=2)),
|
||||
)
|
||||
for operation_id, error_code, updated_at in dead_letter_specs:
|
||||
message = messages[operation_id]
|
||||
message.status = "dead_letter"
|
||||
message.attempt_count = message.max_attempts
|
||||
message.last_error_code = error_code
|
||||
message.last_attempt_at = updated_at - timedelta(seconds=1)
|
||||
message.updated_at = updated_at
|
||||
message.idempotency_key = f"secret-idempotency-{operation_id}"
|
||||
message.payload = {
|
||||
**dict(message.payload),
|
||||
"secret_payload_marker": operation_id,
|
||||
}
|
||||
|
||||
before = governance_storage_snapshot(identity_repository)
|
||||
overview_response = client.get(
|
||||
"/api/v1/identity/privileged-outbox/overview?stale_after_seconds=900",
|
||||
headers=identity_headers(),
|
||||
)
|
||||
first_page_response = client.get(
|
||||
"/api/v1/identity/privileged-outbox/dead-letters?page=1&page_size=2",
|
||||
headers=identity_headers(),
|
||||
)
|
||||
repeated_first_page_response = client.get(
|
||||
"/api/v1/identity/privileged-outbox/dead-letters?page=1&page_size=2",
|
||||
headers=identity_headers(),
|
||||
)
|
||||
second_page_response = client.get(
|
||||
"/api/v1/identity/privileged-outbox/dead-letters?page=2&page_size=2",
|
||||
headers=identity_headers(),
|
||||
)
|
||||
filtered_response = client.get(
|
||||
"/api/v1/identity/privileged-outbox/dead-letters?error_code=transport_error",
|
||||
headers=identity_headers(),
|
||||
)
|
||||
unknown_error_response = client.get(
|
||||
"/api/v1/identity/privileged-outbox/dead-letters?error_code=unknown",
|
||||
headers=identity_headers(),
|
||||
)
|
||||
assert overview_response.status_code == 200
|
||||
overview = overview_response.json()
|
||||
assert overview["health_status"] == "critical"
|
||||
assert overview["status_counts"] == {
|
||||
"queued": 1,
|
||||
"processing": 1,
|
||||
"retry_scheduled": 1,
|
||||
"delivered": 1,
|
||||
"dead_letter": 3,
|
||||
}
|
||||
assert overview["active_count"] == 3
|
||||
assert overview["oldest_active_age_seconds"] >= 900
|
||||
assert overview["stale_active_count"] == 1
|
||||
assert overview["expired_processing_lease_count"] == 1
|
||||
assert overview["near_attempt_limit_count"] == 1
|
||||
assert overview["dead_letter_count"] == 3
|
||||
assert overview["dead_letter_error_codes"] == [
|
||||
{"error_code": "transport_error", "count": 2},
|
||||
{"error_code": "unknown", "count": 1},
|
||||
]
|
||||
|
||||
assert first_page_response.status_code == 200
|
||||
first_page = first_page_response.json()
|
||||
assert first_page == repeated_first_page_response.json()
|
||||
assert first_page["total"] == 3
|
||||
assert first_page["total_pages"] == 2
|
||||
assert len(first_page["items"]) == 2
|
||||
assert [item["message_id"] for item in first_page["items"]] == sorted(
|
||||
[item["message_id"] for item in first_page["items"]],
|
||||
reverse=True,
|
||||
)
|
||||
assert second_page_response.status_code == 200
|
||||
assert len(second_page_response.json()["items"]) == 1
|
||||
assert {
|
||||
item["message_id"]
|
||||
for item in first_page["items"] + second_page_response.json()["items"]
|
||||
} == {
|
||||
item[0]
|
||||
for item in before[2]
|
||||
if item[1].startswith("operation-observe-dead-")
|
||||
}
|
||||
assert filtered_response.status_code == 200
|
||||
filtered = filtered_response.json()
|
||||
assert filtered["total"] == 2
|
||||
assert {item["error_code"] for item in filtered["items"]} == {"transport_error"}
|
||||
assert unknown_error_response.status_code == 200
|
||||
assert unknown_error_response.json()["total"] == 1
|
||||
assert unknown_error_response.json()["items"][0]["error_code"] == "unknown"
|
||||
invalid_error_response = client.get(
|
||||
"/api/v1/identity/privileged-outbox/dead-letters?error_code=unsafe%20detail",
|
||||
headers=identity_headers(),
|
||||
)
|
||||
after = governance_storage_snapshot(identity_repository)
|
||||
assert invalid_error_response.status_code == 422
|
||||
assert invalid_error_response.json()["detail"]["code"] == "invalid_stable_error_code"
|
||||
|
||||
allowed_item_keys = {
|
||||
"message_id",
|
||||
"event_id",
|
||||
"operation_id",
|
||||
"event_type",
|
||||
"operation_type",
|
||||
"effect_type",
|
||||
"trace_id",
|
||||
"attempt_count",
|
||||
"max_attempts",
|
||||
"error_code",
|
||||
"created_at",
|
||||
"updated_at",
|
||||
"last_attempt_at",
|
||||
}
|
||||
for item in first_page["items"] + second_page_response.json()["items"]:
|
||||
assert set(item) == allowed_item_keys
|
||||
assert item["trace_id"].startswith("trace-")
|
||||
serialized = json.dumps(
|
||||
{
|
||||
"overview": overview,
|
||||
"dead_letters": first_page["items"] + second_page_response.json()["items"],
|
||||
},
|
||||
ensure_ascii=True,
|
||||
)
|
||||
for forbidden_value in (
|
||||
"secret_payload_marker",
|
||||
"secret-idempotency",
|
||||
"secret-lease-owner",
|
||||
"target_reference_fingerprints",
|
||||
"actor_account_id",
|
||||
"actor_tenant_id",
|
||||
"actor_user_id",
|
||||
):
|
||||
assert forbidden_value not in serialized
|
||||
assert before == after
|
||||
|
||||
with identity_repository._session_factory.begin() as session:
|
||||
for message in session.execute(
|
||||
select(PrivilegedIdentityOutboxMessageModel).where(
|
||||
PrivilegedIdentityOutboxMessageModel.operation_id.in_(operation_ids)
|
||||
)
|
||||
).scalars().all():
|
||||
if message.status == "dead_letter":
|
||||
message.status = "delivered"
|
||||
message.delivered_at = now
|
||||
message.lease_owner = None
|
||||
elif message.status == "processing":
|
||||
message.lease_expires_at = now + timedelta(minutes=5)
|
||||
|
||||
degraded = client.get(
|
||||
"/api/v1/identity/privileged-outbox/overview?stale_after_seconds=900",
|
||||
headers=identity_headers(),
|
||||
)
|
||||
assert degraded.status_code == 200
|
||||
assert degraded.json()["health_status"] == "degraded"
|
||||
assert degraded.json()["expired_processing_lease_count"] == 0
|
||||
assert degraded.json()["stale_active_count"] == 1
|
||||
assert degraded.json()["near_attempt_limit_count"] == 1
|
||||
|
||||
with identity_repository._session_factory.begin() as session:
|
||||
retry = session.execute(
|
||||
select(PrivilegedIdentityOutboxMessageModel).where(
|
||||
PrivilegedIdentityOutboxMessageModel.operation_id
|
||||
== "operation-observe-retry"
|
||||
)
|
||||
).scalar_one()
|
||||
retry.created_at = datetime.now(UTC)
|
||||
retry.attempt_count = 1
|
||||
healthy = client.get(
|
||||
"/api/v1/identity/privileged-outbox/overview?stale_after_seconds=900",
|
||||
headers=identity_headers(),
|
||||
)
|
||||
assert healthy.status_code == 200
|
||||
assert healthy.json()["health_status"] == "healthy"
|
||||
|
||||
|
||||
def test_outbox_worker_uses_leases_backoff_and_dead_letter(tmp_path: Path) -> None:
|
||||
app = create_app(make_settings(tmp_path))
|
||||
client = TestClient(app)
|
||||
identity_repository = app.state.contract_repository.identity
|
||||
add_identity(
|
||||
identity_repository,
|
||||
account_id="account-governed-target",
|
||||
user_id="user-governed-target",
|
||||
email="governed-target@example.com",
|
||||
platform_admin=False,
|
||||
)
|
||||
preview = grant_preview(client)
|
||||
submit_from_preview(client, preview, operation_id="operation-worker-001")
|
||||
outbox = PrivilegedIdentityOutboxRepository(identity_repository._session_factory)
|
||||
disabled_worker = PrivilegedIdentityOutboxWorker(
|
||||
outbox,
|
||||
worker_id="disabled-worker",
|
||||
)
|
||||
assert disabled_worker.run_once() == 0
|
||||
first = outbox.claim_batch(worker_id="worker-one", batch_size=1, lease_seconds=30)
|
||||
assert len(first) == 1
|
||||
assert first[0].attempt_count == 1
|
||||
assert outbox.mark_delivered(first[0].message_id, worker_id="wrong-worker") is False
|
||||
assert outbox.mark_failed(
|
||||
first[0].message_id,
|
||||
worker_id="worker-one",
|
||||
error_code="TemporaryTransportError",
|
||||
) is True
|
||||
|
||||
with identity_repository._session_factory.begin() as session:
|
||||
model = session.get(PrivilegedIdentityOutboxMessageModel, first[0].message_id)
|
||||
assert model is not None
|
||||
assert model.status == "retry_scheduled"
|
||||
assert model.attempt_count == 1
|
||||
assert model.last_error_code == "temporarytransporterror"
|
||||
assert model.available_at > model.updated_at
|
||||
model.available_at = datetime.now(UTC) - timedelta(seconds=1)
|
||||
model.max_attempts = 2
|
||||
|
||||
second = outbox.claim_batch(worker_id="worker-two", batch_size=1, lease_seconds=30)
|
||||
assert len(second) == 1
|
||||
assert second[0].attempt_count == 2
|
||||
assert outbox.mark_failed(
|
||||
second[0].message_id,
|
||||
worker_id="worker-two",
|
||||
error_code="sensitive error text with spaces",
|
||||
) is True
|
||||
with identity_repository._session_factory() as session:
|
||||
model = session.get(PrivilegedIdentityOutboxMessageModel, second[0].message_id)
|
||||
assert model is not None
|
||||
assert model.status == "dead_letter"
|
||||
assert model.last_error_code == "transport_error"
|
||||
|
||||
|
||||
def test_sqlite_outbox_worker_rejects_a_second_concurrent_worker(tmp_path: Path) -> None:
|
||||
app = create_app(make_settings(tmp_path))
|
||||
client = TestClient(app)
|
||||
identity_repository = app.state.contract_repository.identity
|
||||
add_identity(
|
||||
identity_repository,
|
||||
account_id="account-governed-target",
|
||||
user_id="user-governed-target",
|
||||
email="governed-target@example.com",
|
||||
platform_admin=False,
|
||||
)
|
||||
preview = grant_preview(client)
|
||||
submit_from_preview(client, preview, operation_id="operation-single-worker-001")
|
||||
outbox = PrivilegedIdentityOutboxRepository(identity_repository._session_factory)
|
||||
first_worker = PrivilegedIdentityOutboxWorker(outbox, worker_id="sqlite-worker-one")
|
||||
second_worker = PrivilegedIdentityOutboxWorker(outbox, worker_id="sqlite-worker-two")
|
||||
delivery_started = Event()
|
||||
release_delivery = Event()
|
||||
|
||||
def blocking_delivery(_message) -> None:
|
||||
delivery_started.set()
|
||||
assert release_delivery.wait(timeout=5)
|
||||
|
||||
with ThreadPoolExecutor(max_workers=1) as executor:
|
||||
future = executor.submit(first_worker.run_once, blocking_delivery, batch_size=1)
|
||||
assert delivery_started.wait(timeout=5)
|
||||
with pytest.raises(RuntimeError, match="one worker only"):
|
||||
second_worker.run_once(lambda _message: None, batch_size=1)
|
||||
release_delivery.set()
|
||||
assert future.result(timeout=5) == 1
|
||||
|
||||
|
||||
def test_sqlite_claim_transaction_prevents_duplicate_low_level_claims(tmp_path: Path) -> None:
|
||||
app = create_app(make_settings(tmp_path))
|
||||
client = TestClient(app)
|
||||
identity_repository = app.state.contract_repository.identity
|
||||
add_identity(
|
||||
identity_repository,
|
||||
account_id="account-governed-target",
|
||||
user_id="user-governed-target",
|
||||
email="governed-target@example.com",
|
||||
platform_admin=False,
|
||||
)
|
||||
preview = grant_preview(client)
|
||||
submit_from_preview(client, preview, operation_id="operation-sqlite-claim-lock-001")
|
||||
outbox = PrivilegedIdentityOutboxRepository(identity_repository._session_factory)
|
||||
barrier = Barrier(2)
|
||||
|
||||
def claim(worker_id: str):
|
||||
barrier.wait(timeout=5)
|
||||
return outbox.claim_batch(
|
||||
worker_id=worker_id,
|
||||
batch_size=1,
|
||||
lease_seconds=30,
|
||||
)
|
||||
|
||||
with ThreadPoolExecutor(max_workers=2) as executor:
|
||||
batches = list(executor.map(claim, ["sqlite-claimer-one", "sqlite-claimer-two"]))
|
||||
|
||||
claimed = [message for batch in batches for message in batch]
|
||||
assert len(claimed) == 1
|
||||
assert len({message.message_id for message in claimed}) == 1
|
||||
|
||||
|
||||
def test_expired_outbox_lease_cannot_acknowledge_and_can_be_reclaimed(tmp_path: Path) -> None:
|
||||
app = create_app(make_settings(tmp_path))
|
||||
client = TestClient(app)
|
||||
identity_repository = app.state.contract_repository.identity
|
||||
add_identity(
|
||||
identity_repository,
|
||||
account_id="account-governed-target",
|
||||
user_id="user-governed-target",
|
||||
email="governed-target@example.com",
|
||||
platform_admin=False,
|
||||
)
|
||||
preview = grant_preview(client)
|
||||
submit_from_preview(client, preview, operation_id="operation-lease-expiry-001")
|
||||
outbox = PrivilegedIdentityOutboxRepository(identity_repository._session_factory)
|
||||
claimed_at = datetime.now(UTC)
|
||||
first = outbox.claim_batch(
|
||||
worker_id="lease-worker-one",
|
||||
batch_size=1,
|
||||
lease_seconds=5,
|
||||
now=claimed_at,
|
||||
)
|
||||
assert len(first) == 1
|
||||
after_expiry = claimed_at + timedelta(seconds=6)
|
||||
assert outbox.mark_delivered(
|
||||
first[0].message_id,
|
||||
worker_id="lease-worker-one",
|
||||
now=after_expiry,
|
||||
) is False
|
||||
reclaimed = outbox.claim_batch(
|
||||
worker_id="lease-worker-two",
|
||||
batch_size=1,
|
||||
lease_seconds=5,
|
||||
now=after_expiry,
|
||||
)
|
||||
assert len(reclaimed) == 1
|
||||
assert reclaimed[0].message_id == first[0].message_id
|
||||
assert reclaimed[0].idempotency_key == first[0].idempotency_key
|
||||
assert reclaimed[0].attempt_count == 2
|
||||
assert outbox.mark_delivered(
|
||||
reclaimed[0].message_id,
|
||||
worker_id="lease-worker-two",
|
||||
now=after_expiry + timedelta(seconds=1),
|
||||
) is True
|
||||
|
||||
+257
-2
@@ -25,11 +25,15 @@ from yuqei_contract_api.persistence.models import (
|
||||
PrivilegedIdentityOperationApprovalModel,
|
||||
PrivilegedIdentityOperationEventModel,
|
||||
PrivilegedIdentityOperationModel,
|
||||
PrivilegedIdentityOutboxMessageModel,
|
||||
TenantModel,
|
||||
TenantUserModel,
|
||||
WorkspaceMembershipModel,
|
||||
)
|
||||
from yuqei_contract_api.persistence.privileged_identity import PrivilegedIdentityOperationRepository
|
||||
from yuqei_contract_api.persistence.privileged_identity_outbox import (
|
||||
PrivilegedIdentityOutboxRepository,
|
||||
)
|
||||
from yuqei_contract_api.privileged_identity import (
|
||||
PrivilegedIdentityOperationApprovalRequest,
|
||||
PrivilegedIdentityOperationError,
|
||||
@@ -118,7 +122,7 @@ def postgres_governance() -> Iterator[PostgresGovernanceHarness]:
|
||||
with isolated_engine.connect() as connection:
|
||||
assert connection.scalar(
|
||||
text("SELECT version_num FROM contract_api_alembic_version")
|
||||
) == "20260802_0039"
|
||||
) == "20260803_0041"
|
||||
migrated_tables = set(
|
||||
connection.execute(
|
||||
text("SELECT tablename FROM pg_tables WHERE schemaname = :schema_name"),
|
||||
@@ -131,7 +135,44 @@ def postgres_governance() -> Iterator[PostgresGovernanceHarness]:
|
||||
"privileged_identity_operations",
|
||||
"privileged_identity_operation_approvals",
|
||||
"privileged_identity_operation_events",
|
||||
"privileged_identity_outbox_messages",
|
||||
} <= migrated_tables
|
||||
outbox_constraints = {
|
||||
row[0]
|
||||
for row in connection.execute(
|
||||
text(
|
||||
"""
|
||||
SELECT constraint_name
|
||||
FROM information_schema.table_constraints
|
||||
WHERE table_schema = :schema_name
|
||||
AND table_name = 'privileged_identity_outbox_messages'
|
||||
"""
|
||||
),
|
||||
{"schema_name": schema_name},
|
||||
)
|
||||
}
|
||||
assert {
|
||||
"fk_privileged_identity_outbox_event_operation",
|
||||
"ck_privileged_identity_outbox_lease_state",
|
||||
} <= outbox_constraints
|
||||
event_indexes = {
|
||||
row[0]
|
||||
for row in connection.execute(
|
||||
text(
|
||||
"""
|
||||
SELECT indexname
|
||||
FROM pg_indexes
|
||||
WHERE schemaname = :schema_name
|
||||
AND tablename = 'privileged_identity_operation_events'
|
||||
"""
|
||||
),
|
||||
{"schema_name": schema_name},
|
||||
)
|
||||
}
|
||||
assert (
|
||||
"ux_privileged_identity_operation_events_operation_event"
|
||||
in event_indexes
|
||||
)
|
||||
yield PostgresGovernanceHarness(
|
||||
schema_name=schema_name,
|
||||
session_factory=session_factory,
|
||||
@@ -352,6 +393,46 @@ def _assert_event_integrity(events: list[PrivilegedIdentityOperationEventModel])
|
||||
assert "postgres-concurrency.invalid" not in serialized
|
||||
|
||||
|
||||
def _governance_storage_snapshot(
|
||||
session_factory: sessionmaker[Session],
|
||||
) -> tuple[tuple[tuple[object, ...], ...], ...]:
|
||||
with session_factory() as session:
|
||||
operations = tuple(
|
||||
tuple(row)
|
||||
for row in session.execute(
|
||||
select(*PrivilegedIdentityOperationModel.__table__.c).order_by(
|
||||
PrivilegedIdentityOperationModel.operation_id
|
||||
)
|
||||
).all()
|
||||
)
|
||||
approvals = tuple(
|
||||
tuple(row)
|
||||
for row in session.execute(
|
||||
select(*PrivilegedIdentityOperationApprovalModel.__table__.c).order_by(
|
||||
PrivilegedIdentityOperationApprovalModel.id
|
||||
)
|
||||
).all()
|
||||
)
|
||||
events = tuple(
|
||||
tuple(row)
|
||||
for row in session.execute(
|
||||
select(*PrivilegedIdentityOperationEventModel.__table__.c).order_by(
|
||||
PrivilegedIdentityOperationEventModel.operation_id,
|
||||
PrivilegedIdentityOperationEventModel.sequence_no,
|
||||
)
|
||||
).all()
|
||||
)
|
||||
outbox = tuple(
|
||||
tuple(row)
|
||||
for row in session.execute(
|
||||
select(*PrivilegedIdentityOutboxMessageModel.__table__.c).order_by(
|
||||
PrivilegedIdentityOutboxMessageModel.message_id
|
||||
)
|
||||
).all()
|
||||
)
|
||||
return operations, approvals, events, outbox
|
||||
|
||||
|
||||
def test_postgresql_privileged_governance_concurrency(
|
||||
postgres_governance: PostgresGovernanceHarness,
|
||||
) -> None:
|
||||
@@ -518,6 +599,177 @@ def test_postgresql_privileged_governance_concurrency(
|
||||
assert expiry_events[1].trace_id == "trace-expiry"
|
||||
_assert_event_integrity(expiry_events)
|
||||
|
||||
outbox = PrivilegedIdentityOutboxRepository(session_factory)
|
||||
with session_factory() as session:
|
||||
outbox_count = int(
|
||||
session.scalar(select(func.count(PrivilegedIdentityOutboxMessageModel.message_id)))
|
||||
or 0
|
||||
)
|
||||
event_count = int(
|
||||
session.scalar(select(func.count(PrivilegedIdentityOperationEventModel.event_id)))
|
||||
or 0
|
||||
)
|
||||
stored_messages = session.execute(
|
||||
select(PrivilegedIdentityOutboxMessageModel).order_by(
|
||||
PrivilegedIdentityOutboxMessageModel.created_at,
|
||||
PrivilegedIdentityOutboxMessageModel.message_id,
|
||||
)
|
||||
).scalars().all()
|
||||
assert outbox_count == event_count
|
||||
assert len({message.idempotency_key for message in stored_messages}) == outbox_count
|
||||
for message in stored_messages:
|
||||
assert message.payload_hash == privileged_payload_hash(message.payload)
|
||||
serialized_payload = json.dumps(message.payload, sort_keys=True).lower()
|
||||
for forbidden_value in (
|
||||
"postgres-concurrency.invalid",
|
||||
"user-target",
|
||||
"password",
|
||||
"credential",
|
||||
"secret",
|
||||
"ticket",
|
||||
):
|
||||
assert forbidden_value not in serialized_payload
|
||||
claimed_batches = _concurrent(
|
||||
["trace-outbox-one", "trace-outbox-two"],
|
||||
lambda trace_id: outbox.claim_batch(
|
||||
worker_id=trace_id,
|
||||
batch_size=100,
|
||||
lease_seconds=30,
|
||||
),
|
||||
)
|
||||
claimed = [message for batch in claimed_batches for message in batch]
|
||||
assert len(claimed) == outbox_count
|
||||
assert len({message.message_id for message in claimed}) == outbox_count
|
||||
owner_by_message_id = {
|
||||
message.message_id: trace_id
|
||||
for trace_id, batch in zip(
|
||||
["trace-outbox-one", "trace-outbox-two"],
|
||||
claimed_batches,
|
||||
strict=True,
|
||||
)
|
||||
for message in batch
|
||||
}
|
||||
leased_message = claimed[0]
|
||||
fake_sink_deliveries = [leased_message.idempotency_key]
|
||||
for message in claimed[1:]:
|
||||
assert outbox.mark_delivered(
|
||||
message.message_id,
|
||||
worker_id=owner_by_message_id[message.message_id],
|
||||
)
|
||||
with session_factory.begin() as session:
|
||||
leased_model = session.get(
|
||||
PrivilegedIdentityOutboxMessageModel,
|
||||
leased_message.message_id,
|
||||
)
|
||||
assert leased_model is not None
|
||||
leased_model.lease_expires_at = datetime.now(UTC) - timedelta(seconds=1)
|
||||
recovered = outbox.claim_batch(
|
||||
worker_id="trace-outbox-recovery",
|
||||
batch_size=1,
|
||||
lease_seconds=30,
|
||||
)
|
||||
assert len(recovered) == 1
|
||||
assert recovered[0].message_id == leased_message.message_id
|
||||
assert recovered[0].idempotency_key == leased_message.idempotency_key
|
||||
assert recovered[0].attempt_count == leased_message.attempt_count + 1
|
||||
fake_sink_deliveries.append(recovered[0].idempotency_key)
|
||||
assert fake_sink_deliveries == [leased_message.idempotency_key] * 2
|
||||
assert outbox.mark_failed(
|
||||
recovered[0].message_id,
|
||||
worker_id="trace-outbox-recovery",
|
||||
error_code="temporary_transport_error",
|
||||
)
|
||||
with session_factory.begin() as session:
|
||||
retry_model = session.get(
|
||||
PrivilegedIdentityOutboxMessageModel,
|
||||
leased_message.message_id,
|
||||
)
|
||||
assert retry_model is not None
|
||||
assert retry_model.status == "retry_scheduled"
|
||||
retry_model.available_at = datetime.now(UTC) - timedelta(seconds=1)
|
||||
retry_model.max_attempts = retry_model.attempt_count + 1
|
||||
final_attempt = outbox.claim_batch(
|
||||
worker_id="trace-outbox-final",
|
||||
batch_size=1,
|
||||
lease_seconds=30,
|
||||
)
|
||||
assert len(final_attempt) == 1
|
||||
assert final_attempt[0].idempotency_key == leased_message.idempotency_key
|
||||
assert outbox.mark_failed(
|
||||
final_attempt[0].message_id,
|
||||
worker_id="trace-outbox-final",
|
||||
error_code="temporary_transport_error",
|
||||
)
|
||||
with session_factory() as session:
|
||||
dead_letter = session.get(
|
||||
PrivilegedIdentityOutboxMessageModel,
|
||||
leased_message.message_id,
|
||||
)
|
||||
assert dead_letter is not None
|
||||
assert dead_letter.status == "dead_letter"
|
||||
|
||||
storage_before_observability = _governance_storage_snapshot(session_factory)
|
||||
overview = repository.get_outbox_overview(
|
||||
actor_tenant_id=_TENANT_ID,
|
||||
actor_user_id="user-requester",
|
||||
stale_after_seconds=900,
|
||||
)
|
||||
dead_letter_page = repository.list_outbox_dead_letters(
|
||||
actor_tenant_id=_TENANT_ID,
|
||||
actor_user_id="user-requester",
|
||||
page=1,
|
||||
page_size=20,
|
||||
stable_error_code="temporary_transport_error",
|
||||
)
|
||||
storage_after_observability = _governance_storage_snapshot(session_factory)
|
||||
assert overview.health_status == "critical"
|
||||
assert overview.status_counts.dead_letter == 1
|
||||
assert overview.status_counts.delivered == outbox_count - 1
|
||||
assert overview.active_count == 0
|
||||
assert overview.dead_letter_count == 1
|
||||
assert [item.model_dump() for item in overview.dead_letter_error_codes] == [
|
||||
{"error_code": "temporary_transport_error", "count": 1}
|
||||
]
|
||||
assert dead_letter_page.total == 1
|
||||
assert dead_letter_page.total_pages == 1
|
||||
assert dead_letter_page.items[0].message_id == leased_message.message_id
|
||||
assert dead_letter_page.items[0].error_code == "temporary_transport_error"
|
||||
assert storage_before_observability == storage_after_observability
|
||||
|
||||
foreign_event_id = next(
|
||||
message.event_id
|
||||
for message in stored_messages
|
||||
if message.operation_id != leased_message.operation_id
|
||||
)
|
||||
with pytest.raises(DBAPIError):
|
||||
with session_factory.begin() as session:
|
||||
session.execute(
|
||||
text(
|
||||
"UPDATE privileged_identity_outbox_messages "
|
||||
"SET event_id = :event_id WHERE message_id = :message_id"
|
||||
),
|
||||
{
|
||||
"event_id": foreign_event_id,
|
||||
"message_id": leased_message.message_id,
|
||||
},
|
||||
)
|
||||
for invalid_lease_assignment in (
|
||||
"status = 'processing', lease_owner = NULL, lease_expires_at = NULL",
|
||||
"status = 'processing', lease_owner = 'invalid-worker', lease_expires_at = NULL",
|
||||
"status = 'queued', lease_owner = 'invalid-worker', lease_expires_at = NOW()",
|
||||
):
|
||||
with pytest.raises(DBAPIError):
|
||||
with session_factory.begin() as session:
|
||||
session.execute(
|
||||
text(
|
||||
"UPDATE privileged_identity_outbox_messages "
|
||||
f"SET {invalid_lease_assignment} "
|
||||
"WHERE message_id = :message_id"
|
||||
),
|
||||
{"message_id": leased_message.message_id},
|
||||
)
|
||||
assert storage_before_observability == _governance_storage_snapshot(session_factory)
|
||||
|
||||
with session_factory.begin() as session:
|
||||
users = session.execute(select(TenantUserModel)).scalars().all()
|
||||
for user in users:
|
||||
@@ -550,5 +802,8 @@ def test_postgresql_privileged_governance_concurrency(
|
||||
"same_approver=1 approval/1 decision event; "
|
||||
"distinct_approvers=2 approvals/ordered status approval; "
|
||||
"rejection=terminal/unique; expiry=materialized_once; "
|
||||
"trace_and_snapshot_hashes=verified; final_admin=protected"
|
||||
"trace_and_snapshot_hashes=verified; outbox_skip_locked=unique; "
|
||||
"outbox_lease_retry_dead_letter=verified; "
|
||||
"outbox_observability_read_only=verified; outbox_0041_integrity=verified; "
|
||||
"final_admin=protected"
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user