Files

Production backup and restore verification

Application backup APIs are not infrastructure backups. Production evidence requires PostgreSQL custom dumps, contract-file archives, S3 object exports, checksums, and a disposable database restore rehearsal.

Backup

Dry-run:

python3 yuqei-infra/backup/run-production-backup.py \
  --env-file /etc/yuqei/production.env

Maintenance-window backup:

python3 yuqei-infra/backup/run-production-backup.py \
  --env-file /etc/yuqei/production.env \
  --apply \
  --quiesce-writers \
  --prune-keep 14

An online deployment whose containers retain the immutable repo@sha256:... references needs no additional identity input. An offline BUILD-05 install runs the same candidate by local config digest instead. By default the apply path looks under /var/lib/yuqei/release-package: it accepts exactly one fully valid successful receipt matching the current release, project, frozen candidate environment, and frozen three-layer Compose input. Zero or multiple matching receipts, drift, links, non-private permissions, or a different owner fail closed. An explicit receipt is an operator override, not a required nightly manual step:

python3 yuqei-infra/backup/run-production-backup.py \
  --env-file /etc/yuqei/production.env \
  --runtime-receipt /var/lib/yuqei/release-package/receipts/APPROVED_RECEIPT.json \
  --apply --quiesce-writers --prune-keep 14

printf 'YUQEI_BACKUP_RUNTIME_RECEIPT=/var/lib/yuqei/release-package/receipts/APPROVED_RECEIPT.json\n' \
  | sudo tee /etc/yuqei/backup-runtime.env

The receipt must validate against its frozen execution environment and must bind the current release/project, all candidate manifest references, imported config digests, and the deployed RustFS container. A raw config digest without that candidate binding is rejected. The release-package apply and the timer must run as the same dedicated yuqei-ops account so its 0700 state and 0600/0400 receipts and snapshots remain both readable and private. If a different protected state root is required, set YUQEI_BACKUP_RUNTIME_STATE_DIR in /etc/yuqei/backup-runtime.env; do not copy or loosen the receipt files.

BACKUP_HOST_DIR must normally be a separate NAS/NFS/OSS-mounted filesystem. --allow-local-target exists only for rehearsals and cannot be used as production disaster-recovery evidence.

Each completed directory contains:

  • three PostgreSQL custom-format dumps;
  • contract-files.tar.gz;
  • an object-storage bucket mirror (the minio/ directory name is retained only for package compatibility);
  • a v2 manifest.json with the RustFS engine/client versions and immutable image identities, release ID, sizes, and SHA-256 values.

The backup client is the pinned RustFS rc image and direct client runs use --pull=never. Credentials are omitted from command arguments, container environment, and host temporary files; the runner streams a TOML document over standard input into a private tmpfs-backed client home. The backup container runs as the invoking host UID/GID with a read-only root filesystem, no Linux capabilities, and no-new-privileges.

The v2 manifest, every artifact, the declared image identity, and the running RustFS/client image IDs are verified while the run still has a .backup-*.partial name. Files and directories are flushed before one same-filesystem atomic rename publishes backup-*. Caught failures receive a FAILED marker; an uncatchable host/power loss can leave an unmarked partial directory, but partial directories are never treated or pruned as completed backups.

When writers are quiesced, every writer is restarted independently in a bounded finally path and its running state is checked. A failed restart or SIGTERM/ SIGINT quarantines any completed-name directory back to .partial, writes failure.json, and prevents PRODUCTION_BACKUP_PASS.

Restore check

Checksum and archive verification is read-only:

python3 yuqei-infra/backup/verify-production-backup.py \
  --backup-dir /mnt/yuqei-v2-backups/backup-YYYYMMDDTHHMMSSZ

Disposable database restore rehearsal:

python3 yuqei-infra/backup/verify-production-backup.py \
  --backup-dir /mnt/yuqei-v2-backups/backup-YYYYMMDDTHHMMSSZ \
  --env-file /etc/yuqei/production.env \
  --restore-databases \
  --apply

For RC07, provide --environment-id and --consistency-set-id when they differ from the protected environment value and backup ID. A successful three-database restore of an artifacts-v2 backup also emits consistent-backup-manifest.json in the frozen yuqei-production-backup.v3 shape consumed by the first-release recovery planner. The v3 document carries an external path/SHA-256 record for the raw artifacts-v2 manifest, the application and RustFS/rc runtime identities, and the restored artifact inventory. The planner reopens that original manifest and cross-checks it against the current release package's online references and offline config image IDs. Legacy yuqei-production-backup.v2 documents remain readable for non-writing plans but cannot mint a new install-readiness receipt. The restore-check summary is versioned and self-hashed and binds both the raw artifacts-v2 manifest and that adapter. It deliberately sets independent_restore_test_claimed=false; host snapshot restore and TLS binding still require the separate isolated-UAT evidence expected by RC07.

The script creates only generated yuqei_restore_check_* databases and drops them in finally. It never restores into the configured production database names. Contract-file and object-storage artifacts are integrity-checked but are not written into production volumes by this command. The verifier remains read-compatible with legacy v1 MinIO manifests but new backups write only v2 manifests.

MinIO-to-RustFS volume boundary

RustFS uses the new rustfs-data Compose volume and the production example names it yuqei-v2-production-rustfs-v1. Never point that volume at the historical MinIO volume. The RustFS container checks the volume on every start: it rejects .minio.sys, linked/invalid ownership markers, and an unowned non-empty volume; after the private marker is established, legitimate RustFS data is allowed across restarts. Migration must be a logical S3 export/import with the old MinIO volume retained unchanged for rollback; direct block/file-level reuse is prohibited.

Scheduling

Install yuqei-production-backup.service and .timer under /etc/systemd/system, then use:

systemctl daemon-reload
systemctl enable --now yuqei-production-backup.timer
systemctl list-timers yuqei-production-backup.timer

The default timer uses a short nightly maintenance window. Change that only after validating an online cross-store consistency strategy.