Production backup and restore verification
Application backup APIs are not infrastructure backups. Production evidence requires PostgreSQL custom dumps, contract-file archives, S3 object exports, checksums, and a disposable database restore rehearsal.
Backup
Dry-run:
python3 yuqei-infra/backup/run-production-backup.py \
--env-file /etc/yuqei/production.env
Maintenance-window backup:
python3 yuqei-infra/backup/run-production-backup.py \
--env-file /etc/yuqei/production.env \
--apply \
--quiesce-writers \
--prune-keep 14
An online deployment whose containers retain the immutable repo@sha256:...
references needs no additional identity input. An offline BUILD-05 install runs
the same candidate by local config digest instead. By default the apply path
looks under /var/lib/yuqei/release-package: it accepts exactly one fully valid
successful receipt matching the current release, project, frozen candidate
environment, and frozen three-layer Compose input. Zero or multiple matching
receipts, drift, links, non-private permissions, or a different owner fail
closed. An explicit receipt is an operator override, not a required nightly
manual step:
python3 yuqei-infra/backup/run-production-backup.py \
--env-file /etc/yuqei/production.env \
--runtime-receipt /var/lib/yuqei/release-package/receipts/APPROVED_RECEIPT.json \
--apply --quiesce-writers --prune-keep 14
printf 'YUQEI_BACKUP_RUNTIME_RECEIPT=/var/lib/yuqei/release-package/receipts/APPROVED_RECEIPT.json\n' \
| sudo tee /etc/yuqei/backup-runtime.env
The receipt must validate against its frozen execution environment and must
bind the current release/project, all candidate manifest references, imported
config digests, and the deployed RustFS container. A raw config digest without
that candidate binding is rejected. The release-package apply and the timer
must run as the same dedicated yuqei-ops account so its 0700 state and
0600/0400 receipts and snapshots remain both readable and private. If a
different protected state root is required, set
YUQEI_BACKUP_RUNTIME_STATE_DIR in /etc/yuqei/backup-runtime.env; do not copy
or loosen the receipt files.
BACKUP_HOST_DIR must normally be a separate NAS/NFS/OSS-mounted filesystem. --allow-local-target exists only for rehearsals and cannot be used as production disaster-recovery evidence.
Each completed directory contains:
- three PostgreSQL custom-format dumps;
contract-files.tar.gz;- an object-storage bucket mirror (the
minio/directory name is retained only for package compatibility); - a v2
manifest.jsonwith the RustFS engine/client versions and immutable image identities, release ID, sizes, and SHA-256 values.
The backup client is the pinned RustFS rc image and direct client runs use
--pull=never. Credentials are omitted from command arguments, container
environment, and host temporary files; the runner streams a TOML document over
standard input into a private tmpfs-backed client home. The backup container
runs as the invoking host UID/GID with a read-only root filesystem, no Linux
capabilities, and no-new-privileges.
The v2 manifest, every artifact, the declared image identity, and the running RustFS/client image IDs are verified while the run still has a .backup-*.partial name. Files and directories are flushed before one same-filesystem atomic rename publishes backup-*. Caught failures receive a FAILED marker; an uncatchable host/power loss can leave an unmarked partial directory, but partial directories are never treated or pruned as completed backups.
When writers are quiesced, every writer is restarted independently in a bounded
finally path and its running state is checked. A failed restart or SIGTERM/
SIGINT quarantines any completed-name directory back to .partial, writes
failure.json, and prevents PRODUCTION_BACKUP_PASS.
Restore check
Checksum and archive verification is read-only:
python3 yuqei-infra/backup/verify-production-backup.py \
--backup-dir /mnt/yuqei-v2-backups/backup-YYYYMMDDTHHMMSSZ
Disposable database restore rehearsal:
python3 yuqei-infra/backup/verify-production-backup.py \
--backup-dir /mnt/yuqei-v2-backups/backup-YYYYMMDDTHHMMSSZ \
--env-file /etc/yuqei/production.env \
--restore-databases \
--apply
For RC07, provide --environment-id and --consistency-set-id when they differ
from the protected environment value and backup ID. A successful three-database
restore of an artifacts-v2 backup also emits
consistent-backup-manifest.json in the frozen
yuqei-production-backup.v3 shape consumed by the first-release recovery
planner. The v3 document carries an external path/SHA-256 record for the raw
artifacts-v2 manifest, the application and RustFS/rc runtime identities, and the
restored artifact inventory. The planner reopens that original manifest and
cross-checks it against the current release package's online references and
offline config image IDs. Legacy yuqei-production-backup.v2 documents remain
readable for non-writing plans but cannot mint a new install-readiness receipt.
The restore-check summary is versioned and self-hashed and binds both the raw
artifacts-v2 manifest and that adapter. It deliberately sets
independent_restore_test_claimed=false; host snapshot restore and TLS binding
still require the separate isolated-UAT evidence expected by RC07.
The script creates only generated yuqei_restore_check_* databases and drops them in finally. It never restores into the configured production database names. Contract-file and object-storage artifacts are integrity-checked but are not written into production volumes by this command. The verifier remains read-compatible with legacy v1 MinIO manifests but new backups write only v2 manifests.
MinIO-to-RustFS volume boundary
RustFS uses the new rustfs-data Compose volume and the production example names it yuqei-v2-production-rustfs-v1. Never point that volume at the historical MinIO volume. The RustFS container checks the volume on every start: it rejects .minio.sys, linked/invalid ownership markers, and an unowned non-empty volume; after the private marker is established, legitimate RustFS data is allowed across restarts. Migration must be a logical S3 export/import with the old MinIO volume retained unchanged for rollback; direct block/file-level reuse is prohibited.
Scheduling
Install yuqei-production-backup.service and .timer under /etc/systemd/system, then use:
systemctl daemon-reload
systemctl enable --now yuqei-production-backup.timer
systemctl list-timers yuqei-production-backup.timer
The default timer uses a short nightly maintenance window. Change that only after validating an online cross-store consistency strategy.