python3 yuqei-infra/observability/check-production-health.py \
  --edge-base-url https://api.contracts.example.com \
  --contract-api-base http://127.0.0.1:8100 \
  --contract-proxy-key-file /etc/yuqei/monitoring/contract-proxy-key \
  --ai-api-base http://127.0.0.1:8101 \
  --ai-service-key-file /etc/yuqei/secrets/ai-platform-service-key \
  --backup-root /mnt/yuqei-v2-backups \
  --backup-max-age-hours 30 \
  --root-disk-path / \
  --home-disk-path /home \
  --home-require-separate-mount \
  --backup-disk-path /mnt/yuqei-v2-backups \
  --backup-require-separate-mount \
  --webhook-url-file /etc/yuqei/secrets/alert-webhook-url

Install the included systemd service and timer for one-minute checks. Production preflight requires PRODUCTION_ALERT_WEBHOOK_URL_FILE to point to an absolute, owner-only file containing the approved HTTPS receiver URL. The included unit passes that file to every one-minute probe; do not put the URL or credentials in the repository or directly in the environment file.

The production environment template fixes warning/critical ceilings at 85%/90% used and free-space floors at 20/10 GiB for each monitored filesystem. A path is a warning or failure when either boundary is crossed. Missing paths and required paths that are not separate mounts fail closed.

Warning and failure summaries are posted to the configured webhook when one is provided. A warning still exits 0 after successful delivery, a failed health check exits 1, and an attempted webhook delivery failure exits 2 so systemd records the alerting failure. Every run writes production-health-summary.json, including the exact capacity policy and alert-delivery result, under project-docs/qa-artifacts/v2-production-health/.

The installed service deduplicates an unchanged failing check set for one hour. The first failure, any failed/warning check-set change, and the hourly reminder are delivered. A failed delivery is retried on the next one-minute run, and a healthy run resets the state so a recurrence is delivered immediately. The state is written atomically beside the health artifacts and contains no webhook URL or credential.

The production unit uses the host's /usr/bin/python3.11 and runs as the unprivileged nobody account so NFS backup manifests remain readable under the export's root-squash policy.

Acceptance commands (they do not start services or change traffic):

python3 -m pytest yuqei-infra/tests/test_observability_capacity.py -q
systemd-analyze verify \
  yuqei-infra/observability/yuqei-production-health.service \
  yuqei-infra/observability/yuqei-production-health.timer

After deployment, alert acceptance must use a temporary command-line threshold below current usage and an isolated artifact directory. Do not fill a filesystem or edit the installed timer to inject a warning. Retain both the generated summary (attempted=true, delivered=true) and the receiver-side event ID.

receive-production-alert.py and yuqei-production-alert-receiver.service provide a minimal self-hosted receiver when an approved third-party receiver is not available. Run it as the dedicated yuqei-alert user on a separate node, bind it to loopback, and expose only POST /ops/alerts/<random-token> through an existing HTTPS reverse proxy. Keep the token in owner-restricted files on both nodes. Each accepted alert is written as a mode 0600 JSON receipt containing a unique event ID, receipt time, source, request-body SHA-256, and the exact health payload. Do not expose the receipt directory or add a public read endpoint.

This probe is a minimum launch guard, not a replacement for centralized logs, long-term metrics, PostgreSQL slow-query analysis, and infrastructure dashboards.