ai-contract-platform/project-docs/V2-production-rollback-capacity-audit-2026-08-12.md

V2 production rollback and capacity audit

Date: 2026-08-12

Scope and safety boundary

This audit began with repository tests and read-only SSH checks, then used an authorized maintenance window for backup/restore verification and a complete application rollback/forward rehearsal. The public edge remained stopped. PostgreSQL, Redis and MinIO were not rolled back, no database downgrade was run, and DNS/public traffic were not changed.

Result

The primary-host capacity policy is currently within its configured limits, but it has little percentage headroom: the latest production health artifact reported / at 84.45% used with 28.90 GiB free, immediately below the 85% warning ceiling. /home was 16.94% used with 192.14 GiB free, and the off-host backup mount was 14.42% used with 68.24 GiB free. Both required paths were observed as separate mounts by the production health probe.

Capacity code and repository-level alert behavior pass their automated tests. The production health unit wires the protected PRODUCTION_ALERT_WEBHOOK_URL_FILE into the probe, and production preflight fails closed unless that file contains an owner-restricted HTTPS URL. A receiver on the independent host 47.94.178.212 accepted the controlled warning and returned a receiver-side event ID, receipt time and body hash. The normal 85% threshold was restored and reported / at 84.45%, status pass.

The production timer remains enabled while the edge is intentionally stopped, so the edge check is expected to fail closed. To avoid sending the identical known failure every minute, the probe now retains an atomic, non-secret alert state: the first failure, any check-set change and an hourly reminder are sent; failed delivery retries on the next run; a healthy run resets the state.

A complete application version rollback and forward restoration was rehearsed from current release d6a6752258a555f3a40894cdcf88917659538492 to approved previous release c24f7a4f1ee041d92feca96bf147e423e58da17c, then back to the current release. All four application services were stopped/recreated in both directions. The Contract API and worker digests were identical between those releases; the AI API image changed. This limitation is recorded rather than misrepresenting the exercise as four distinct image rollbacks. Health, schemas, worker readiness, contract/template reads and audit/outbox continuity passed in both directions. The alert and complete version rollback gates are closed.

Read-only field evidence

Primary host: 192.168.3.90 (host-reported name 10.20.0.11).

Observed production services:

• Contract API, Contract review worker, AI Platform API, AI worker, PostgreSQL, Redis and MinIO were running and healthy.
• Contract API was pinned to git.yuqei.com/yuqei/ai-contract-platform-contract-api@sha256:a6969d90680f866232f1ad2c09607fd63f226476028b6c0f34c3175e7407549f.
• AI Platform API was pinned to git.yuqei.com/yuqei/ai-contract-platform-ai-api@sha256:05faf2aa67d258708387daa7007442928b8d786814f1c5e8a619db4b1ae9d8b3.
• The latest normal-policy production health artifact had status fail only because edge_health returned HTTP 503. Contract/AI health, schema checks, Provider runtime, worker health, backup freshness and all three capacity checks passed.
• A controlled capacity warning was received as event yuqei-alert-45b05755b2fc4c498dd4eb8cd3c426ed; its posted body SHA-256 was 933d5741e600efec3190e79892a93165bb94a57f00976184fde31ec43148eeff.
• The receiver-owned mode 0600 receipt SHA-256 was 8816696a8829215ce7e7821a0965d35d48a27651c89b1a267846d84bd664f909.
• Backup backup-20260812T025650Z completed in quiesced mode with four artifacts. Disposable restoration of all three PostgreSQL databases passed; summary SHA-256 was 70e6c223a6870dd0d8fe28365bda7f883fc6827a77ddcc0da397dd77cbee4370.
• /etc/yuqei/production.env existed with mode 0600.
• yuqei-production-health.timer was enabled and active at a one-minute cadence.
• /etc/yuqei/monitoring/alert-webhook-url is configured without exposing its secret URL in repository evidence. It is mode 0600, owned by the unprivileged probe account. The receiver is loopback-bound behind HTTPS.
• After deploying deduplication, the first expected edge_health failure was delivered as yuqei-alert-9014cb51c85241f5b63836bf8cb1a510; an immediate second identical run recorded duplicate_within_resend_window, did not call the receiver, left the timer active and kept the edge replica count at 0.
• Current production read-only preflight passed with failed=0. Its retained summary SHA-256 is 3b1e2e3b25272adf4c1443fb98cbe7284b520d82808836f879de58c4e4240556.

Complete rollback and forward evidence

• Remote evidence directory: /opt/apps/ai-contract-platform-v2-production/project-docs/qa-artifacts/v2-production-full-rollback/20260812T030852Z.
• Manifest SHA-256: 2ac616acc69868a8274ab0047d42dcdcd2b8ff29660ac68486fcbdc007d1d4a4.
• Evidence file count: 40; manifest status: passed.
• Before, rollback and forward phases all passed Contract health/schema, AI health/schema, worker health, contract-list read, template-list read and audit/outbox count continuity.
• The edge replica count remained 0. Provider readiness remained blocked by the intentional kill switch and, on the previous release, missing receipt binding. This is expected fail-closed behavior, not a failed recovery check.
• An exit trap restored the current environment on any failure. Final inspection confirmed the current release and all four application services healthy.

Independent public-policy trust evidence

• Legal approval ID: policy-approval-0887bbd5149bf7a1c41cb35e.
• Trust request SHA-256: d2d76ee0789b235af67f1df3ae3a86237467ca72bbf76ad6a613e35a0f996eee.
• Independent operations reviewer: Chen Yong, stable approver ID chenyong@yuqei.com, key ID policy-trust-ops-2026q3.
• Independent public-key file SHA-256: 0e2b147e5da9d1084da368944c720821a053b866282f1c5f9385dbad529e4009.
• Exact trust payload SHA-256: 1b783cb6a9f1976d50f7d8b4c77e286425d7c8bec7c597057a172b6eaa113442.
• Signed trust approval ID: policy-key-trust-approval-09d9860e94d6df9fe74a0d1e; approval SHA-256: 5093462cefff6a75e5f9a320f8bd4e033d9327e0234f84e9bbc258c36148fbbc.
• Trusted policy keyring SHA-256: 41f9517dcd375afbeea4aaa4dce5d8bbe07db8f71e2706d47e7aec5ccd8bd388.
• Strict verification at 2026-08-12T12:40:02.047339+08:00 passed for all four exact policy files without candidate-keyring override.
• The signed decision binds the out-of-band identity/authority and legal-key fingerprint attestations. policy_activation_authorized=false; the trusted keyring remains local and has not been installed in production. No policy was published or activated by this approval.

Host filesystem observations were also checked directly with df -P: root was 85% at df display precision, while /home was 17%. The application probe's non-reserved-capacity calculation is the authoritative threshold comparison and reported root at 84.45%.

Repository verification

Executed on 2026-08-12:

Command	Result
python -m pytest yuqei-infra/tests/test_observability_capacity.py yuqei-infra/tests/test_alert_receiver.py -q	26 passed
python -m pytest yuqei-infra/tests -q	148 passed
python -m py_compile on all changed Python entrypoints	passed
python -m bandit -q on all changed Python entrypoints	passed
git diff --check	passed; only existing line-ending warnings

The capacity tests cover warning/critical percent and free-space thresholds, reserved filesystem blocks, separate-mount fail-closed behavior, webhook success, webhook timeout, receiver authentication and immutable receipts, healthy no-alert behavior, changed-state and one-hour resend deduplication, production preflight thresholds, and the installed systemd arguments.

Capacity and alert operating policy

1. Keep the HTTPS URL in its owner-restricted file; never put its secret path or token in the repository, environment file or audit report.
2. Retain the receiver-side receipt, local summary and event ID for each acceptance test.
3. Keep one-hour unchanged-alert deduplication enabled. Do not suppress the first failure, a changed check set, delivery retries or hourly reminders.
4. Treat root at or above 85%, free below 20 GiB, or more than five percentage points growth in 24 hours as a release stop. Do not delete business volumes, Registry data or backups to satisfy the gate. Move rebuildable caches/logs only after an explicit inventory and retention approval, or extend the root LV.

Complete version rollback acceptance policy

The completed exercise followed these controls. Re-run it after a material release topology, image or migration change; do not infer future downgrade compatibility from this one result.

1. Record the current release ID, all image digests, receipt/key fingerprints, Alembic heads, container health and golden-smoke results.
2. Take and verify a new off-host production backup. Restore all three dumps into disposable yuqei_restore_check_* databases before continuing.
3. Prove migration compatibility. If the previous application cannot read the current schema, stop: use forward-fix recovery unless a separately reviewed, data-preserving downgrade exists. Never infer downgrade safety from an Alembic command succeeding in an empty database.
4. With edge and public traffic still disabled, switch application services only to the approved previous digests. Do not roll back PostgreSQL/Redis/MinIO data or delete audit/outbox records.
5. Verify health, schema checks, immutable release fingerprints, login/workspace, contract list/detail, template drafting, review read path and audit continuity.
6. Return to the current immutable digests, re-run migrations in the forward direction, and repeat the same checks.
7. Capture start/end times, RTO, every command exit code, before/rollback/forward fingerprints, database heads, release-gate summaries and operator approval.
8. Abort immediately on migration drift, partial health, missing audit/outbox data, receipt mismatch, unexpected writes, or any need to use an unpinned image.

Acceptance requires both rollback and forward-restoration checks to pass, the database and immutable audit history to remain intact, and an operator-owned evidence package. The 2026-08-12 rehearsal satisfies this gate for the recorded release pair.

Gate status

Gate	Status	Reason
Capacity thresholds and live primary-host readings	pass, narrow headroom	root 84.45% used; warning begins at 85%
Capacity timer and local alert contract	pass	timer active; automated warning/failure delivery semantics pass
Real external alert delivery	pass	HTTPS receiver event and immutable receipt hashes captured
Backup integrity and disposable DB restore	pass from existing field evidence	three dumps and restore checks completed
Complete production version rollback and forward restore	pass	40-file evidence manifest passed for previous/current release pair
Independent public-policy trust approval	pass, policy not activated	Chen Yong signed the exact trust payload as operations reviewer; strict verification issued a non-activating trusted keyring
Public traffic authorization	not in scope	edge/DNS/traffic remain separately controlled