系统备份容灾与回滚审计
检查关键数据备份、恢复演练与部署回退能力,形成恢复缺口清单和整改方案。
完整提示词
# BACKUP, DISASTER RECOVERY AND ROLLBACK AUDIT I want an ultimate forensic audit of backup, restore, disaster recovery, and rollback capabilities across the complete system. Main objective: > Determine whether the system can truly recover from data deletion, corruption, ransomware/admin compromise, bad migrations, broken deployments, region/provider failures, secret losses, and infrastructure destruction, rather than merely assuming that "backups exist." Most critical rule: > A backup that has never been restored is not proof of recoverability. ## 1. INVENTORY CRITICAL STATE - primary database - object/file storage - user uploads - configs - secrets - encryption keys - queue state - search index - analytics - source artifacts - deployment artifacts - DNS/IaC state ## 2. CLASSIFY For each asset: ```text Authoritative: Rebuildable: Backup required: Recovery method: ``` ## 3. DATABASE BACKUP Verify: - full backup - incremental - WAL/binlog - snapshots - PITR ## 4. FREQUENCY Do not guess expected RPO. If no requirement is specified: **RPO NOT DEFINED** ## 5. RETENTION How far back can the system be restored? ## 6. CORRUPTION WINDOW If corruption goes undetected for 30 days while retention is 7 days: backup does not solve the issue. ## 7. PITR Granularity and retention limits. ## 8. RESTORE TEST When was the last actual restore exercise conducted? ## 9. RESTORE IN ISOLATION Never restore over production as a test. ## 10. DATA VALIDATION A successful restore command output is insufficient. Verify: - tables - row counts - constraints - application login - critical reads/writes ## 11. ENCRYPTED BACKUP Where is the encryption key stored? ## 12. LOST KEY An encrypted backup without the corresponding key is unusable. ## 13. KEY BACKUP However, the key must not reside unprotected alongside the backup. ## 14. ACCESS Who has permission to read backups? ## 15. DELETE Who has permission to delete backups? ## 16. SAME CREDENTIAL If a single compromised admin credential can delete: ```text production + all backups ``` the blast radius is catastrophic. ## 17. IMMUTABILITY Object lock/WORM storage where justified. ## 18. SEPARATE ACCOUNT/PROJECT For critical backup threat models. Do not demand universally without justification. ## 19. OFFSITE Same physical or provider failure domain. ## 20. CROSS-REGION Only if a regional disaster recovery requirement exists. ## 21. FILE STORAGE Database backups do not automatically incorporate uploaded files. ## 22. OBJECT STORAGE VERSIONING Recovery from accidental overwrite or deletion. ## 23. LIFECYCLE Version expiration schedules. ## 24. CDN Is not a backup solution. ## 25. CACHE If disposable, rebuild. If holding authoritative state, backup and recovery required. ## 26. REDIS Sessions/queues/locks vs durable business data. ## 27. QUEUE Does losing queued messages equate to irrecoverable business data loss? ## 28. DLQ Is not a backup. ## 29. SEARCH INDEX Can it be reconstructed from the primary database? ## 30. ANALYTICS Is analytics data critical to business continuity? ## 31. SECRET MANAGER How to recover: - deleted secret - rotated key - lost encryption key ## 32. SIGNING KEY Loss can force global user logouts or completely break token verification. ## 33. PRIVATE TLS KEY Certificate reissue procedure. ## 34. CODE SIGNING KEY Loss or compromise response plan. ## 35. IaC STATE Terraform state backup and versioning. ## 36. IaC SOURCE Can rebuild infrastructure, but cannot restore data. ## 37. DNS Zone file exports and configuration backups. ## 38. DEPLOYMENT ARTIFACT Previous production artifact availability. ## 39. REGISTRY RETENTION Ensure rollback container images are not pruned. ## 40. PACKAGE ARTIFACT Desktop and mobile release archives. ## 41. DATABASE MIGRATION Backing up prior to a destructive migration is insufficient if restore takes 12h while business RTO is 15min. ## 42. FORWARD FIX VS RESTORE Restores can occasionally cause greater loss of recently committed data than forward fixes. ## 43. SELECTIVE RESTORE Can a single tenant, table, or object be recovered? ## 44. FULL RESTORE Complete system recovery. ## 45. LOGICAL BACKUP Useful for granular and selective recovery. ## 46. PHYSICAL SNAPSHOT Fast but coarse and provider-dependent. ## 47. CONSISTENCY Across multiple datastores: ```text DB + object storage ``` backup timestamps can be out of sync. ## 48. APPLICATION-CONSISTENT BACKUP When required. ## 49. TRANSACTIONAL LINK Database record pointing to specific object store versions. ## 50. PARTIAL RESTORE Database restored to yesterday while object storage remains current. ## 51. CROSS-SYSTEM RPO The slowest or retention-limited state dictates the actual recovery point. ## 52. DISASTER SCENARIOS Model separately: - accidental delete - malicious delete - corruption - bad migration - broken deploy - ransomware/admin compromise - provider outage - region outage - account lockout - secret compromise - key loss ## 53. ACCIDENTAL DELETE The most frequent real-world recovery scenario. ## 54. SOFT DELETE Is not a backup. ## 55. MALICIOUS DELETE An attacker can permanently purge soft-deleted records. ## 56. BAD SCRIPT Bulk database update corruption. ## 57. BUG CORRUPTION Backups taken after a silent bug was introduced will preserve corrupted state. ## 58. DETECTION TIME How quickly is silent corruption discovered? ## 59. BACKUP RETENTION > DETECTION WINDOW If not, a permanent recovery gap exists. ## 60. RANSOMWARE/CLOUD ADMIN Assume production credentials are fully compromised. ## 61. BACKUP CREDENTIAL SEPARATION Can an attacker pivot to the backup storage plane? ## 62. PROVIDER ACCOUNT LOSS Can the organization regain access to the hosting account? Operational and business continuity planning. ## 63. REGION LOSS Only evaluate if multi-region is an explicit disaster target. ## 64. MULTI-REGION DATA Replication is not a backup. ## 65. REPLICATION Corrupted data or deletions replicate immediately. ## 66. HA ≠ BACKUP Crucial distinction. ## 67. BACKUP ≠ HA Equally crucial. ## 68. DR ≠ ROLLBACK Keep concepts distinct. ## 69. APPLICATION ROLLBACK Redeploying the previous artifact. ## 70. DATABASE ROLLBACK Often impossible following destructive schema modifications. ## 71. CONFIG ROLLBACK Restoring previous configuration versions. ## 72. SECRET ROLLBACK Never restore a known-compromised secret. ## 73. INFRA ROLLBACK IaC previous state redeployment. ## 74. FEATURE FLAG Can avoid full binary rollbacks. ## 75. DEPLOYMENT ROLLBACK Must be tested in practice. ## 76. ROLLBACK AFTER MIGRATION Critical verification point. ## 77. ROLLBACK AFTER NEW DATA FORMAT The older application might not know how to parse data written by the newer release. ## 78. FORWARD COMPATIBILITY Compatibility requirements. ## 79. QUEUE ROLLBACK Old workers attempting to read new message payloads. ## 80. CACHE ROLLBACK Serialized format mismatches. ## 81. RESTORE DURATION Measure empirically, do not guess. ## 82. RTO If undefined: **RTO NOT DEFINED** ## 83. RESTORE BOTTLENECK - download - decompression - DB import - object restore - DNS - validation ## 84. COLD STORAGE Retrieval delays and warming times. ## 85. LARGE DB Restore durations grow non-linearly. ## 86. PARALLEL RESTORE Potential optimization, not an automatic assumption. ## 87. RUNBOOK Step-by-step procedural runbooks. ## 88. OWNER Who has designated ownership of the restore procedure? ## 89. ACCESS DURING INCIDENT Do incident responders maintain access if the primary IAM provider is impaired? ## 90. BREAK-GLASS Procedures for emergency break-glass recovery. ## 91. CONTACTS Provider escalation contacts and support tiers. ## 92. EVIDENCE PRESERVATION Incident restores should not prematurely destroy forensic logs or evidence. ## 93. CLEAN RESTORE If attacker persistence exists, restoring a compromised image or config reintroduces the attacker. ## 94. SECRET ROTATION AFTER RESTORE Mandatory rotation following security compromise. ## 95. DR EXERCISE Tabletop review vs actual technical restore execution. ## 96. TABLETOP Useful for coordination, but not technical proof of recoverability. ## 97. GAME DAY Controlled disaster simulation drills. ## 98. RESTORE AUTOMATION Reduces manual operator error during crises. ## 99. AUTOMATION BUG Recovery automation scripts must themselves be regularly tested. ## 100. BACKUP MONITORING Alert on: - failed backup - stale backup - size anomaly - PITR disabled - replication issue ## 101. "SUCCESS" CHECK A 0-byte backup job can exit with code 0 if payload validation is omitted. ## 102. SIZE TREND Valuable metric for anomaly detection. ## 103. RESTORE CHECKSUM Cryptographic integrity verification. ## 104. BACKUP CATALOG Maintain an accurate registry of existing backups. ## 105. OWNERSHIP Designated owners for each recovery domain. ## 106. DOCUMENTATION DRIFT Runbooks pointing to deprecated services, outdated paths, or stale credentials. ## 107. DEPENDENCY RESTORE Third-party SaaS data may require dedicated export or backup processes. ## 108. EMAIL/CRM/EXTERNAL SYSTEM Only if authoritative business data resides exclusively on external platforms. ## 109. USER EXPORT Is not an adequate substitute for system-wide backups. ## 110. LEGAL RETENTION Do not make legal assertions unless regulatory specifications are provided. ## 111. DATA DELETION REQUIREMENTS Compliance-mandated deletions can conflict with immutable backup retention. ## 112. BACKUP SECURITY Backups are often more lucrative targets for attackers than production databases due to unmonitored offline access. ## 113. PASSWORD HASHES Sensitive exposure risks. ## 114. TOKENS Older backups may contain unexpired API or refresh tokens. ## 115. SECRET ROTATION + OLD BACKUP Restoring an older snapshot can resurrect rotated credentials or stale security state. ## 116. RESTORE SANITIZATION Ensure revoked or compromised tokens remain revoked following a database restore. ## 117. USER PASSWORD CHANGE Point-in-time recovery prior to a user password reset will revert to the old hash. Security implications. ## 118. ACCOUNT DELETION Restores can resurrect accounts deleted under privacy mandates. Business and privacy handling requirements. ## 119. AUDIT LOG Restores must preserve or reconcile chronological audit events. ## 120. ID GENERATORS Rolling back database sequences can cause primary key collisions with external systems. ## 121. EXTERNAL SIDE EFFECTS You cannot roll back dispatched emails or processed credit card payments. ## 122. PAYMENT RECONCILIATION Database restores may drop records of transactions finalized by the payment gateway. ## 123. WEBHOOK REPLAY Provider webhook events must be reconciled post-restore. ## 124. EVENT SOURCING If event sourcing is utilized, the recovery model differs significantly. ## 125. THIRD-PARTY SOURCE OF TRUTH Establish which system holds authoritative state. ## 126. DR MODE Operating in a degraded, read-only mode during recovery. ## 127. MAINTENANCE PAGE Preserving user experience and preventing erroneous writes during restoration. ## 128. DNS FAILOVER If an alternate standby environment exists. ## 129. COLD STANDBY Cost vs RTO trade-off. ## 130. WARM STANDBY Warm standby considerations. ## 131. HOT STANDBY Do not recommend automatically without cost justification. ## 132. REGION RESTORE Can IaC plus offsite backups reconstruct an entire region? ## 133. ACCOUNT-LEVEL DISASTER Separate account backup strategies apply primarily to high-assurance threat models. ## 134. SINGLE PROVIDER Not inherently unacceptable for all architectures. ## 135. FINDING FORMAT ```text ID: Severity: Status: Evidence tier: Asset: Authoritative: Failure scenario: Backup mechanism: Frequency: Retention: Restore tested: Last known restore: RPO requirement: Actual estimated RPO: RTO requirement: Measured/estimated RTO: Failure path: Data loss: Availability impact: Security impact: Evidence: Root cause: Remediation: Restore verification: ``` ## 136. SEVERITY P0: - critical data has no usable recovery and realistic catastrophic loss path exists - backup encryption/key loss makes all recovery impossible - one compromised credential can irreversibly destroy production + all backups and no secondary recovery exists P1: - backup exists but proven unusable - actual RPO/RTO grossly violates explicit critical requirement - rollback procedure can further corrupt/destroy production - region/account failure has no recovery despite explicit DR requirement P2: - meaningful restore gap - partial data class omitted - stale/unverified runbook - insufficient retention vs realistic corruption detection P3: - limited recovery weakness P4: - maturity/hardening ## 137. EVIDENCE ```text A - successful/failed restore exercise or production incident evidence B - complete backup/restore config and path C - strong config evidence D - inferred E - maturity ``` ## 138. STATUS AND FALSE POSITIVES Status: - **CONFIRMED** - tier A or B evidence shows the failure or exploit path. - **LIKELY** - tier C evidence. - **NOT VERIFIED** - depends on runtime state, settings or versions that could not be checked (tier D). Never present tier D as confirmed. - **NOT APPLICABLE** - the component or pattern is not used. - **CONTROLLED** - the risk exists but another control contains it. - **HARDENING** - improvement without a current failure path (P4). Recoverability is CONFIRMED only by a restore exercise or incident evidence (tier A); a complete configuration (tier B) proves that the backup path exists, not that the restore works within RPO/RTO. False-positive rules: - A single region or single provider is not a defect when the stated recovery requirements do not demand more. - Missing point-in-time recovery is a finding only when a realistic corruption or deletion scenario needs a finer restore point than existing backups provide. - Derived or rebuildable state (caches, search indexes, analytics copies) does not need backups if the rebuild path and its duration are acceptable. - Retention shorter than a compliance guideline is a finding only when a requirement applies or realistic detection time exceeds it. - An irreversible migration is not a rollback defect if a tested roll-forward path exists. Do not report a missing best practice as a confirmed defect unless there is a concrete failure, exploit, correctness, reliability, or operational path. ## 139. OUTPUT `BACKUP_DISASTER_RECOVERY_ROLLBACK_AUDIT.md` ## 140. MATRICES ### Critical State Matrix | State | Authoritative | Backup | PITR | Restore tested | |---|---|---|---|---| ### Failure Matrix | Disaster | Data affected | Recovery | RPO | RTO | |---|---|---|---|---| ### Rollback Matrix | Component | Previous version available | Data compatible | Rollback tested | |---|---|---|---| ### Credential Separation | Identity | Prod delete | Backup read | Backup delete | KMS | |---|---|---|---|---| ## 141. SECOND PASS - RESTORE EXERCISES Mandatory review: ### Scenario A Accidental deletion of a single table or critical resource. ### Scenario B Bad migration corrupts active data. ### Scenario C Production database completely destroyed. ### Scenario D Object storage files or buckets deleted. ### Scenario E Production cloud administrator account compromised. ### Scenario F Backup encryption key unavailable or destroyed. ### Scenario G Latest backup corrupted upon restore. ### Scenario H Restoration required to a state from 24 hours ago. ### Scenario I Application rollback required following a database migration. ### Scenario J Complete cloud region or provider outage (if within scope). For each scenario: ```text Detection ↓ Decision ↓ Containment ↓ Restore ↓ Validation ↓ Traffic return ↓ Reconciliation ``` ## 142. SECOND PASS - ACTUAL RESTORE If a safe, isolated staging/test environment is available: 1. obtain a production-like backup 2. restore into a blank, isolated environment 3. launch the application stack 4. validate end-to-end data integrity 5. record actual elapsed recovery time 6. document all errors encountered Never restore over production purely for audit validation. ## 143. SECOND PASS - CREDENTIAL COMPROMISE Assume an attacker gains root/admin credentials in production. Ask: > Can the attacker delete backups, destroy KMS keys, and purge audit logs using those same credentials? ## 144. SECOND PASS - LONG-LIVED CORRUPTION Assume a bug silently corrupts data for 30 days before detection. Does existing retention allow recovery to an uncorrupted snapshot? ## 145. SECOND PASS - ROLLBACK REALITY Deployment sequence: ```text v1 -> migration -> v2 ``` then evaluate conceptually or in staging: ```text v2 -> v1 ``` Does v1 still understand: - schema - new rows - enum values - queue messages - cache - config ## 146. FINAL QUALITY GATE Prior to final submission, verify: - every authoritative datastore - file and object storage - secrets and encryption keys - deployment artifacts - backup storage location - access and deletion permissions - PITR configuration - retention windows - encryption key recovery procedures - verified restore tests - measured/estimated restore duration - explicit RPO/RTO requirements - application validation following restore - cross-store data consistency - rollback feasibility following migrations - queue, cache, and client compatibility - attacker/admin compromise blast radius - backup monitoring and alerting - runbook ownership and freshness - external side-effect reconciliation # FINAL RULE Looking for issues such as: ```text database backup: daily ↓ backup encrypted with customer-managed key ↓ same cloud admin identity can: delete DB delete backups schedule KMS key deletion ↓ admin credential compromised ↓ production + backups + decryption capability lost ↓ recovery impossible ``` or: ```text migration adds new enum values ↓ v2 writes new values ↓ deployment later fails ↓ team rolls back to v1 ↓ v1 cannot deserialize new enum ↓ rollback deployment itself causes outage ``` or: ```text DB is restored to yesterday ↓ payment provider was not rolled back ↓ provider has 500 successful payments ↓ restored DB remembers only 450 ↓ system may retry/reconcile incorrectly ↓ financial state divergence ``` If backups exist but restore was never verified: **RECOVERABILITY NOT VERIFIED.** If RPO/RTO are not explicitly defined: **REQUIREMENT NOT DEFINED.** If only DR maturity improvements are identified without an empirical gap: **P4 - HARDENING.**
分类与标签
编程开发、备份恢复、容灾演练、部署回滚、恢复目标