ai-contract-platform/project-docs/V2-production-rollback-capacity-audit-2026-08-12.md

# V2 production rollback and capacity audit

Date: 2026-08-12

## Scope and safety boundary

This audit began with repository tests and read-only SSH checks, then used an
authorized maintenance window for backup/restore verification and a complete
application rollback/forward rehearsal. The public edge remained stopped.
PostgreSQL, Redis and MinIO were not rolled back, no database downgrade was run,
and DNS/public traffic were not changed.

## Result

The primary-host capacity policy is currently within its configured limits,
but it has little percentage headroom: the latest production health artifact
reported `/` at 84.45% used with 28.90 GiB free, immediately below the 85%
warning ceiling. `/home` was 16.94% used with 192.14 GiB free, and the off-host
backup mount was 14.42% used with 68.24 GiB free. Both required paths were
observed as separate mounts by the production health probe.

Capacity code and repository-level alert behavior pass their automated tests.
The production health unit wires the protected
`PRODUCTION_ALERT_WEBHOOK_URL_FILE` into the probe, and production preflight
fails closed unless that file contains an owner-restricted HTTPS URL. A receiver
on the independent host `47.94.178.212` accepted the controlled warning and
returned a receiver-side event ID, receipt time and body hash. The normal 85%
threshold was restored and reported `/` at 84.45%, status `pass`.

The production timer remains enabled while the edge is intentionally stopped,
so the edge check is expected to fail closed. To avoid sending the identical
known failure every minute, the probe now retains an atomic, non-secret alert
state: the first failure, any check-set change and an hourly reminder are sent;
failed delivery retries on the next run; a healthy run resets the state.

A complete application version rollback and forward restoration was rehearsed
from current release `d6a6752258a555f3a40894cdcf88917659538492` to approved
previous release `c24f7a4f1ee041d92feca96bf147e423e58da17c`, then back to the
current release. All four application services were stopped/recreated in both
directions. The Contract API and worker digests were identical between those
releases; the AI API image changed. This limitation is recorded rather than
misrepresenting the exercise as four distinct image rollbacks. Health, schemas,
worker readiness, contract/template reads and audit/outbox continuity passed in
both directions. The alert and complete version rollback gates are **closed**.

## Read-only field evidence

Primary host: `192.168.3.90` (host-reported name `10.20.0.11`).

Observed production services:

- Contract API, Contract review worker, AI Platform API, AI worker, PostgreSQL,
  Redis and MinIO were running and healthy.
- Contract API was pinned to
  `git.yuqei.com/yuqei/ai-contract-platform-contract-api@sha256:a6969d90680f866232f1ad2c09607fd63f226476028b6c0f34c3175e7407549f`.
- AI Platform API was pinned to
  `git.yuqei.com/yuqei/ai-contract-platform-ai-api@sha256:05faf2aa67d258708387daa7007442928b8d786814f1c5e8a619db4b1ae9d8b3`.
- The latest normal-policy production health artifact had status `fail` only
  because `edge_health` returned HTTP 503. Contract/AI health, schema checks,
  Provider runtime, worker health, backup freshness and all three capacity
  checks passed.
- A controlled capacity warning was received as event
  `yuqei-alert-45b05755b2fc4c498dd4eb8cd3c426ed`; its posted body SHA-256 was
  `933d5741e600efec3190e79892a93165bb94a57f00976184fde31ec43148eeff`.
- The receiver-owned mode `0600` receipt SHA-256 was
  `8816696a8829215ce7e7821a0965d35d48a27651c89b1a267846d84bd664f909`.
- Backup `backup-20260812T025650Z` completed in quiesced mode with four
  artifacts. Disposable restoration of all three PostgreSQL databases passed;
  summary SHA-256 was
  `70e6c223a6870dd0d8fe28365bda7f883fc6827a77ddcc0da397dd77cbee4370`.
- `/etc/yuqei/production.env` existed with mode `0600`.
- `yuqei-production-health.timer` was enabled and active at a one-minute cadence.
- `/etc/yuqei/monitoring/alert-webhook-url` is configured without exposing its
  secret URL in repository evidence. It is mode `0600`, owned by the unprivileged
  probe account. The receiver is loopback-bound behind HTTPS.
- After deploying deduplication, the first expected `edge_health` failure was
  delivered as `yuqei-alert-9014cb51c85241f5b63836bf8cb1a510`; an immediate
  second identical run recorded `duplicate_within_resend_window`, did not call
  the receiver, left the timer active and kept the edge replica count at `0`.
- Current production read-only preflight passed with `failed=0`. Its retained
  summary SHA-256 is
  `3b1e2e3b25272adf4c1443fb98cbe7284b520d82808836f879de58c4e4240556`.

## Complete rollback and forward evidence

- Remote evidence directory:
  `/opt/apps/ai-contract-platform-v2-production/project-docs/qa-artifacts/v2-production-full-rollback/20260812T030852Z`.
- Manifest SHA-256:
  `2ac616acc69868a8274ab0047d42dcdcd2b8ff29660ac68486fcbdc007d1d4a4`.
- Evidence file count: `40`; manifest status: `passed`.
- Before, rollback and forward phases all passed Contract health/schema, AI
  health/schema, worker health, contract-list read, template-list read and
  audit/outbox count continuity.
- The edge replica count remained `0`. Provider readiness remained blocked by
  the intentional kill switch and, on the previous release, missing receipt
  binding. This is expected fail-closed behavior, not a failed recovery check.
- An exit trap restored the current environment on any failure. Final inspection
  confirmed the current release and all four application services healthy.

## Independent public-policy trust evidence

- Legal approval ID: `policy-approval-0887bbd5149bf7a1c41cb35e`.
- Trust request SHA-256:
  `d2d76ee0789b235af67f1df3ae3a86237467ca72bbf76ad6a613e35a0f996eee`.
- Independent operations reviewer: Chen Yong, stable approver ID
  `chenyong@yuqei.com`, key ID `policy-trust-ops-2026q3`.
- Independent public-key file SHA-256:
  `0e2b147e5da9d1084da368944c720821a053b866282f1c5f9385dbad529e4009`.
- Exact trust payload SHA-256:
  `1b783cb6a9f1976d50f7d8b4c77e286425d7c8bec7c597057a172b6eaa113442`.
- Signed trust approval ID:
  `policy-key-trust-approval-09d9860e94d6df9fe74a0d1e`; approval
  SHA-256:
  `5093462cefff6a75e5f9a320f8bd4e033d9327e0234f84e9bbc258c36148fbbc`.
- Trusted policy keyring SHA-256:
  `41f9517dcd375afbeea4aaa4dce5d8bbe07db8f71e2706d47e7aec5ccd8bd388`.
- Strict verification at `2026-08-12T12:40:02.047339+08:00` passed for all
  four exact policy files without candidate-keyring override.
- The signed decision binds the out-of-band identity/authority and legal-key
  fingerprint attestations. `policy_activation_authorized=false`; the trusted
  keyring remains local and has not been installed in production. No policy was
  published or activated by this approval.

Host filesystem observations were also checked directly with `df -P`: root was
85% at `df` display precision, while `/home` was 17%. The application probe's
non-reserved-capacity calculation is the authoritative threshold comparison and
reported root at 84.45%.

## Repository verification

Executed on 2026-08-12:

| Command | Result |
| --- | --- |
| `python -m pytest yuqei-infra/tests/test_observability_capacity.py yuqei-infra/tests/test_alert_receiver.py -q` | `26 passed` |
| `python -m pytest yuqei-infra/tests -q` | `148 passed` |
| `python -m py_compile` on all changed Python entrypoints | passed |
| `python -m bandit -q` on all changed Python entrypoints | passed |
| `git diff --check` | passed; only existing line-ending warnings |

The capacity tests cover warning/critical percent and free-space thresholds,
reserved filesystem blocks, separate-mount fail-closed behavior, webhook success,
webhook timeout, receiver authentication and immutable receipts, healthy
no-alert behavior, changed-state and one-hour resend deduplication, production
preflight thresholds, and the installed systemd arguments.

## Capacity and alert operating policy

1. Keep the HTTPS URL in its owner-restricted file; never put its secret path or
   token in the repository, environment file or audit report.
2. Retain the receiver-side receipt, local summary and event ID for each
   acceptance test.
3. Keep one-hour unchanged-alert deduplication enabled. Do not suppress the first
   failure, a changed check set, delivery retries or hourly reminders.
4. Treat root at or above 85%, free below 20 GiB, or more than five percentage
   points growth in 24 hours as a release stop. Do not delete business volumes,
   Registry data or backups to satisfy the gate. Move rebuildable caches/logs only
   after an explicit inventory and retention approval, or extend the root LV.

## Complete version rollback acceptance policy

The completed exercise followed these controls. Re-run it after a material
release topology, image or migration change; do not infer future downgrade
compatibility from this one result.

1. Record the current release ID, all image digests, receipt/key fingerprints,
   Alembic heads, container health and golden-smoke results.
2. Take and verify a new off-host production backup. Restore all three dumps into
   disposable `yuqei_restore_check_*` databases before continuing.
3. Prove migration compatibility. If the previous application cannot read the
   current schema, stop: use forward-fix recovery unless a separately reviewed,
   data-preserving downgrade exists. Never infer downgrade safety from an Alembic
   command succeeding in an empty database.
4. With edge and public traffic still disabled, switch application services only
   to the approved previous digests. Do not roll back PostgreSQL/Redis/MinIO data
   or delete audit/outbox records.
5. Verify health, schema checks, immutable release fingerprints, login/workspace,
   contract list/detail, template drafting, review read path and audit continuity.
6. Return to the current immutable digests, re-run migrations in the forward
   direction, and repeat the same checks.
7. Capture start/end times, RTO, every command exit code, before/rollback/forward
   fingerprints, database heads, release-gate summaries and operator approval.
8. Abort immediately on migration drift, partial health, missing audit/outbox data,
   receipt mismatch, unexpected writes, or any need to use an unpinned image.

Acceptance requires both rollback and forward-restoration checks to pass, the
database and immutable audit history to remain intact, and an operator-owned
evidence package. The 2026-08-12 rehearsal satisfies this gate for the recorded
release pair.

## Gate status

| Gate | Status | Reason |
| --- | --- | --- |
| Capacity thresholds and live primary-host readings | pass, narrow headroom | root 84.45% used; warning begins at 85% |
| Capacity timer and local alert contract | pass | timer active; automated warning/failure delivery semantics pass |
| Real external alert delivery | pass | HTTPS receiver event and immutable receipt hashes captured |
| Backup integrity and disposable DB restore | pass from existing field evidence | three dumps and restore checks completed |
| Complete production version rollback and forward restore | pass | 40-file evidence manifest passed for previous/current release pair |
| Independent public-policy trust approval | pass, policy not activated | Chen Yong signed the exact trust payload as operations reviewer; strict verification issued a non-activating trusted keyring |
| Public traffic authorization | not in scope | edge/DNS/traffic remain separately controlled |