yuqei
6c79a998cb
Build and Test Quality Gate / Contract Web typecheck and build (pull_request) Successful in 11m11s
Build and Test Quality Gate / Python tests and Contract API migration (pull_request) Failing after 27m49s
Security Quality Gate / npm audit, Bandit, pip-audit, detect-secrets (pull_request) Successful in 11m13s
62 lines
2.6 KiB
Markdown
62 lines
2.6 KiB
Markdown
# Production minimum monitoring
|
|
|
|
`check-production-health.py` is the first-release operational guard. It checks:
|
|
|
|
- public HTTPS edge health;
|
|
- Contract API health and schema-check;
|
|
- AI Platform health, schema-check, and `provider_runtime`;
|
|
- worker runtime health;
|
|
- latest production backup age;
|
|
- root, `/home`, and backup-mount warning and critical thresholds;
|
|
- both used-percent ceilings and remaining-free-space floors;
|
|
- separate-mount enforcement for `/home` and the off-host backup path.
|
|
|
|
Example:
|
|
|
|
```bash
|
|
python3 yuqei-infra/observability/check-production-health.py \
|
|
--edge-base-url https://api.contracts.example.com \
|
|
--contract-api-base http://127.0.0.1:8100 \
|
|
--contract-proxy-key-file /etc/yuqei/monitoring/contract-proxy-key \
|
|
--ai-api-base http://127.0.0.1:8101 \
|
|
--ai-service-key-file /etc/yuqei/secrets/ai-platform-service-key \
|
|
--backup-root /mnt/yuqei-v2-backups \
|
|
--backup-max-age-hours 30 \
|
|
--root-disk-path / \
|
|
--home-disk-path /home \
|
|
--home-require-separate-mount \
|
|
--backup-disk-path /mnt/yuqei-v2-backups \
|
|
--backup-require-separate-mount \
|
|
--webhook-url-file /etc/yuqei/secrets/alert-webhook-url
|
|
```
|
|
|
|
Install the included systemd service and timer for one-minute checks. The webhook file is optional but, when configured, must contain an HTTPS URL and mode `0600`.
|
|
|
|
The production environment template fixes warning/critical ceilings at 85%/90%
|
|
used and free-space floors at 20/10 GiB for each monitored filesystem. A path
|
|
is a warning or failure when either boundary is crossed. Missing paths and
|
|
required paths that are not separate mounts fail closed.
|
|
|
|
Warning and failure summaries are posted to the configured webhook when one is
|
|
provided. A warning still exits `0` after successful delivery, a failed health
|
|
check exits `1`, and an attempted webhook delivery failure exits `2` so
|
|
systemd records the alerting failure. Every run writes
|
|
`production-health-summary.json`, including the exact capacity policy and
|
|
alert-delivery result, under
|
|
`project-docs/qa-artifacts/v2-production-health/`.
|
|
|
|
The production unit uses the host's `/usr/bin/python3.11` and runs as the
|
|
unprivileged `nobody` account so NFS backup manifests remain readable under the
|
|
export's root-squash policy.
|
|
|
|
Acceptance commands (they do not start services or change traffic):
|
|
|
|
```bash
|
|
python3 -m pytest yuqei-infra/tests/test_observability_capacity.py -q
|
|
systemd-analyze verify \
|
|
yuqei-infra/observability/yuqei-production-health.service \
|
|
yuqei-infra/observability/yuqei-production-health.timer
|
|
```
|
|
|
|
This probe is a minimum launch guard, not a replacement for centralized logs, long-term metrics, PostgreSQL slow-query analysis, and infrastructure dashboards.
|