开展DevOps与基础设施生产审计
梳理部署、运行环境、网络、存储和恢复流程,定位具体生产风险并形成审计结论。
完整提示词
# ULTIMATE DEVOPS AND INFRASTRUCTURE AUDIT I want you to perform a maximally deep, systematic, evidence-first, and production-oriented analysis of the entire DevOps, cloud, and infrastructure layer of the project. Main objective: > Determine whether deployment, infrastructure, environment configuration, compute, networking, secrets, storage, DNS, observability, backups, autoscaling, release processes, and operational procedures can cause outages, data loss, security incidents, deployment failures, environment drift, rollback failures, or severe production degradation. This is not: - a generic DevOps checklist - automatically demanding Kubernetes - automatically demanding Terraform - automatically demanding microservices - automatically switching to multi-region - automatically scaling replicas - automatically adding Redis - merely a CI/CD audit - merely a cloud security audit - merely a cost optimization audit - advising that everything should be "containerized" - a blanket demand for 99.999% availability The focus is on the entire production chain: ```text source ↓ build ↓ artifact ↓ configuration ↓ deployment ↓ runtime ↓ network ↓ dependencies ↓ storage/data ↓ observability ↓ recovery ``` Priority: **data loss > production outage > security boundary failure > irreversible deployment > configuration drift > shared infrastructure bottleneck > recovery failure > observability gaps > cost inefficiency > hardening** It is better to identify 10 genuine production failure paths than to write 200 generic cloud recommendations. --- # 1. ESTABLISH ACTUAL INFRASTRUCTURE Prior to generating findings, map: - hosting platform - cloud provider - regions - compute model - containers - serverless - VMs - Kubernetes - managed services - database - cache - queue - object storage - CDN - load balancer - DNS - CI/CD - secret storage - monitoring - backups If something is not accessible: **NOT VERIFIED** --- # 2. DO NOT INVENT COMPONENTS If the project uses Vercel + managed Postgres: do not draft Kubernetes recommendations without justification. If it utilizes a single VM: analyze that specific architecture. --- # 3. BUILD AN INFRASTRUCTURE MAP Example: ```text Internet ↓ DNS ↓ CDN / Edge ↓ Load Balancer ↓ Application Instances ↓ Database ↓ Cache ↓ Queue ↓ Workers ↓ Object Storage ``` Incorporate genuine components. --- # 4. INVENTORY ENVIRONMENTS Map: ```text development test preview staging production ``` --- # 5. ENVIRONMENT ISOLATION For each environment verify: - credentials - database - storage - queues - domains - provider accounts - secrets --- # 6. STAGING -> PRODUCTION PIVOT Look for shared: - signing secret - DB credentials - storage key - API key - service account --- # 7. PREVIEW ENVIRONMENT Specifically verify whether preview deployments receive production secrets. --- # 8. ENVIRONMENT NAMING Do not rely solely on: ```text NODE_ENV ``` if the platform employs additional environment concepts. --- # 9. CONFIGURATION SOURCE Determine where configuration originates: - env - config files - secret manager - database - platform settings - runtime flags --- # 10. CONFIG PRECEDENCE Example: ```text defaults ↓ config file ↓ environment ↓ CLI ``` Must be unambiguously defined. --- # 11. DEFAULT CONFIG Look for unsafe fallbacks: ```text DB_SSL=false AUTH_DISABLED=true SECRET=changeme ``` --- # 12. MISSING CONFIG Critical configuration must not silently receive insecure default values. --- # 13. FAIL FAST If production lacks: - DB URL - signing secret - encryption key the application should fail loudly and immediately where appropriate. --- # 14. CONFIG DRIFT Compare: ```text repo CI staging production ``` --- # 15. MANUAL CLICKOPS Manual cloud modifications are not automatically an issue. However, they can introduce drift that remains invisible in the repository. --- # 16. INFRASTRUCTURE AS CODE If present: - Terraform - Pulumi - CloudFormation - Bicep - CDK - Helm - Kubernetes YAML audit the actual state against the declared state where feasible. --- # 17. IaC IS NOT MANDATORY Do not report the absence of Terraform as a vulnerability. --- # 18. STATE FILE If using Terraform: verify: - storage - access - locking - secrets --- # 19. DRIFT DETECTION If IaC exists: the question is whether manual modifications remain undetected. --- # 20. COMPUTE MODEL Determine: ```text VM container serverless Kubernetes pod PaaS instance edge function ``` --- # 21. INSTANCE LIFECYCLE Can an instance terminate or vanish at any moment? If yes: local state is not durable. --- # 22. LOCAL DISK Check whether the application retains: - uploads - jobs - sessions - cache - generated files on an ephemeral filesystem. --- # 23. EPHEMERAL FILESYSTEM Serverless/PaaS restarts can destroy data. --- # 24. INSTANCE RESTART Simulate: ```text kill instance ``` in a test environment. Ask what is lost. --- # 25. STATELESSNESS Horizontal scaling is simpler if application state is decoupled from individual instances. --- # 26. STICKY SESSION If present: verify: - failover - uneven load distribution - session loss --- # 27. IN-MEMORY SESSION A restart can log out active users. This may be acceptable or severe depending on product requirements. --- # 28. IN-MEMORY JOB If critical work exists exclusively in RAM: an instance crash results in lost jobs. --- # 29. LOCAL CRON Horizontally scaled replicas may execute the same cron task multiple times. --- # 30. SINGLE INSTANCE ASSUMPTION Search code and configuration for logic presuming: ```text only one server exists ``` --- # 31. APPLICATION REPLICAS Determine: - min - max - desired - autoscaling --- # 32. SINGLE INSTANCE PRODUCTION Not automatically a flaw. However, it entails: - no compute redundancy - restart downtime Evaluate against SLO and business requirements. --- # 33. LOAD BALANCER If present: verify: - health checks - connection behavior - session affinity - timeout --- # 34. HEALTH CHECK A health endpoint must not simply: ```text return 200 ``` if the application is not actually functional. --- # 35. LIVENESS VS READINESS If the platform differentiates between: - liveness - readiness verify their individual semantics. --- # 36. READINESS A new instance must not receive inbound traffic prior to: - config load - migrations dependency readiness - critical initialization --- # 37. LIVENESS LOOP A defective liveness check can restart a slow but healthy service, causing a crash loop. --- # 38. DEPENDENCY HEALTH Do not probe every downstream dependency on every liveness ping if that makes the health check a source of outages. --- # 39. STARTUP Map: ```text process starts ↓ config ↓ DB connect ↓ cache ↓ migrations? ↓ warmup ↓ ready ``` --- # 40. MIGRATION ON STARTUP If every replica runs migrations concurrently: race conditions and lock contention arise. --- # 41. MIGRATION RUNNER Determine whether migrations are executed by: - each application instance - a distinct deployment step - a singleton job --- # 42. DESTRUCTIVE MIGRATION Look for: - DROP - rename - type change - not-null - data rewrite --- # 43. ZERO-DOWNTIME COMPATIBILITY Old and new application versions may run concurrently for brief intervals. The database schema must tolerate a mixed-version period where rollouts operate that way. --- # 44. EXPAND-CONTRACT For breaking schema modifications evaluate the necessity of: ```text expand deploy migrate contract ``` --- # 45. LARGE TABLE MIGRATION A query taking 2 seconds in development can lock a production table for hours. --- # 46. BACKFILL Do not execute massive backfills as blocking migrations without prior analysis. --- # 47. MIGRATION ROLLBACK Ask: > Does rolling back the application code work following a schema migration? --- # 48. IRREVERSIBLE MIGRATION If not: the deployment must have a specialized recovery plan. --- # 49. DEPLOYMENT STRATEGY Determine: - rolling - blue/green - canary - recreate - serverless atomic-like deploy --- # 50. ROLLING DEPLOY Mixed versions in flight. --- # 51. BLUE/GREEN Ask: - data compatibility - background workers - duplicated jobs --- # 52. CANARY Useful only if metrics and automated rollback decision criteria exist. --- # 53. ATOMIC DEPLOYMENT Frontend and backend may not be deployed simultaneously. --- # 54. CLIENT/BACKEND COMPATIBILITY Old browser or mobile clients may remain active for extended periods. --- # 55. API BACKWARD COMPATIBILITY Deployments must not abruptly break existing clients without intent. --- # 56. FEATURE FLAG Can decouple code deployment from feature activation. --- # 57. FEATURE FLAG FAILURE A stale or missing flag must not trigger an unsafe default state. --- # 58. DARK LAUNCH Can mitigate rollout risk for resource-intensive features. Do not prescribe generically. --- # 59. ROLLBACK Must be genuinely viable, not merely: ```text git revert ``` --- # 60. ARTIFACT ROLLBACK Does the previous immutable artifact remain accessible? --- # 61. CONFIG ROLLBACK An older application binary may require older configuration values. --- # 62. DATABASE ROLLBACK Frequently the most complex recovery challenge. --- # 63. EXTERNAL SIDE EFFECT A deployment rollback cannot undo: - sent emails - payment transactions - data deletions --- # 64. FAILED DEPLOY Ask what happens if deployment halts at 50%. --- # 65. MIXED FLEET Half old version, half new version. --- # 66. JOB WORKER VERSION Queues may contain jobs emitted by older code that must be processed by newer code. --- # 67. JOB SCHEMA COMPATIBILITY Payload versioning where required. --- # 68. SCHEDULED JOB DURING DEPLOY Can execute old and new logic concurrently. --- # 69. CONNECTION DRAINING An instance undergoing termination must finish or cleanly terminate: - HTTP requests - jobs - WebSockets --- # 70. GRACEFUL SHUTDOWN Audit: - SIGTERM - server close - queue consumption stop - timeout --- # 71. HARD KILL What occurs if the host platform forcibly terminates the process? Critical operations must have recovery mechanisms. --- # 72. LONG REQUEST A deployment can interrupt: - uploads - exports - streams - payments --- # 73. WEBSOCKET DEPLOY Connections will be severed. Evaluate client reconnection behavior. --- # 74. DATABASE Determine: - managed vs self-hosted - primary - replicas - backups - failover --- # 75. SINGLE DB Scaling application replicas does not provide database high availability. --- # 76. DB MULTI-AZ If a managed service advertises HA: verify the actual provisioned tier. --- # 77. DB FAILOVER Ask: - connection interruption - DNS propagation - reconnection logic - in-flight transaction outcome --- # 78. CONNECTION POOL Total: ```text instances × pool size ``` must fit within database connection limits. --- # 79. AUTOSCALING CONNECTION STORM Application scale-out can exhaust database connections and crash the database. --- # 80. SERVERLESS + DB High numbers of concurrent functions directly querying databases is high-risk. --- # 81. CONNECTION PROXY If present: verify semantics; do not assume it resolves all connection issues. --- # 82. LONG TRANSACTION Can stall failover, block migrations, and hold locks indefinitely. --- # 83. READ REPLICA If present: verify replication lag. --- # 84. READ-AFTER-WRITE Critical user flows may need to read directly from the primary. --- # 85. BACKUP Do not merely ask: > Does a backup exist? Ask: > Has a restore ever been successfully tested? --- # 86. BACKUP FREQUENCY Do not invent an RPO. Derive it from business requirements if defined. --- # 87. RETENTION Map retention policies. --- # 88. OFFSITE / SEPARATE FAILURE DOMAIN A backup stored on the identical host or cloud account shares failure modes. --- # 89. BACKUP CREDENTIAL Ransomware or compromised cloud admin credentials can delete backups as well. --- # 90. IMMUTABLE BACKUP May be warranted for mission-critical data. Do not prescribe universally. --- # 91. RESTORE TEST Document: ```text latest tested restore restore duration data validation ``` if accessible. --- # 92. BACKUP WITHOUT RESTORE TEST This represents only assumed recoverability. --- # 93. RPO If product requirements exist: compare backup and log replication against them. If none exist: **RPO REQUIREMENT NOT DEFINED** --- # 94. RTO Likewise. --- # 95. POINT-IN-TIME RECOVERY If the database engine supports it: verify whether it is enabled and configured. --- # 96. ACCIDENTAL DELETE Recovery scenario. --- # 97. BAD MIGRATION Recovery scenario. --- # 98. APPLICATION BUG CORRUPTION Backups may contain already corrupted records. --- # 99. BACKUP VERSION HISTORY Required to restore state before the corruption window began. --- # 100. OBJECT STORAGE Audit: - durability - ACL - versioning - lifecycle - deletion --- # 101. PUBLIC BUCKET Security audit cross-reference. --- # 102. VERSIONING Can aid recovery from accidental overwrites or deletions. --- # 103. LIFECYCLE RULE Must not prematurely delete critical data. --- # 104. STORAGE CLASS Cold archive storage can have extended restore retrieval times. Factor into RTO. --- # 105. LOCAL FILE BACKUP If the application utilizes local persistent volumes: are they backed up? --- # 106. CACHE Determine whether the cache is: - disposable - authoritative - session store - lock store - queue --- # 107. CACHE FAILURE If Redis crashes: what happens? --- # 108. CACHE AS DATABASE If the only copy of critical state exists inside the cache: high-risk. --- # 109. CACHE PERSISTENCE If Redis employs persistent storage: verify recovery semantics. --- # 110. CACHE COLD START The database must be able to withstand the cache miss storm. --- # 111. QUEUE Audit: - durability - retry - DLQ - capacity - worker deployment --- # 112. QUEUE FAILURE Producers: - fail - buffer - drop ? --- # 113. MESSAGE DURABILITY If the message broker restarts: do messages survive? --- # 114. QUEUE STORAGE Disk capacity and retention policies. --- # 115. DLQ A dead-letter queue is insufficient if nobody monitors or replays it. --- # 116. POISON MESSAGE A single malformed message must not permanently stall the consumer. --- # 117. QUEUE BACKLOG Operational alerting must track: - depth - oldest age --- # 118. WORKER AUTOSCALING Do not scale workers beyond downstream capacity limits. --- # 119. EXTERNAL SERVICES Inventory: - payments - email - SMS - auth - AI - storage - analytics --- # 120. HARD DEPENDENCY Ask: > If this service fails, does the entire application collapse? --- # 121. OPTIONAL DEPENDENCY Analytics failure should not block checkout if not business-critical. --- # 122. TIMEOUT Every remote call must enforce bounded waiting. --- # 123. RETRY Must be operation-aware. --- # 124. CIRCUIT BREAKER Useful for select dependencies, not universally. --- # 125. PROVIDER QUOTA An autoscaling application does not automatically expand upstream provider quotas. --- # 126. DNS Audit: - authoritative provider - records - TTL - ownership - stale records --- # 127. DANGLING DNS An old CNAME pointing to a deleted cloud resource can represent subdomain takeover risk. --- # 128. DOMAIN EXPIRY Operational and security crown jewel. --- # 129. DNSSEC P4 hardening based on threat model. Do not demand automatically. --- # 130. TLS Determine where TLS terminates. --- # 131. CERTIFICATE RENEWAL Automated vs manual. --- # 132. CERT EXPIRY A simple yet catastrophic outage scenario. --- # 133. CUSTOM DOMAIN Platform-level certificate automation may suffice. --- # 134. ORIGIN TLS If edge-to-origin traffic traverses public networks: verify transport trust. --- # 135. LOAD BALANCER TLS Correct cert/SNI/hostname. --- # 136. NETWORK EXPOSURE Inventory public ports and exposed services. --- # 137. DATABASE PUBLIC INTERNET May be acceptable under strict controls, but warrants high-scrutiny review. --- # 138. CACHE PUBLIC INTERNET Usually dangerous. --- # 139. MANAGEMENT PORT Debug/metrics/admin ports. --- # 140. FIREWALL / SECURITY GROUP Verify actual rules. --- # 141. `0.0.0.0/0` Not automatically a bug if the port is meant to be public. Context is essential. --- # 142. EGRESS Application outbound access can be excessively broad. Relevant for SSRF and post-compromise blast radius. --- # 143. EGRESS RESTRICTION Advanced hardening, not a blanket demand. --- # 144. PRIVATE NETWORK Do not assume internal equals safe. --- # 145. SERVICE DISCOVERY Stale endpoints and failure semantics. --- # 146. NAT Outbound connection limits and port exhaustion under high concurrency. --- # 147. IPV4 / IPV6 Security rules may protect IPv4 while neglecting IPv6. --- # 148. CDN Audit: - cache rules - origin exposure - authenticated content - invalidation --- # 149. PRIVATE RESPONSE CACHE Critical data leakage risk. --- # 150. ORIGIN BYPASS CDN, WAF, and rate-limiting controls can be bypassed via direct origin access. --- # 151. WAF Do not treat as a root security control for application bugs. --- # 152. RATE LIMIT Edge limits combined with origin reachability. --- # 153. SERVERLESS If relevant: audit: - execution timeout - memory - concurrency - cold start - connection limits - ephemeral storage --- # 154. SERVERLESS MAX DURATION Background work exceeding function timeout limits requires an alternative execution model. --- # 155. SERVERLESS RETRY The platform may automatically retry asynchronous events. --- # 156. SERVERLESS CONCURRENCY Bursts can overwhelm databases and upstream providers. --- # 157. COLD START Latency spike plus dependency initialization storm. --- # 158. FUNCTION ENV Secrets and configuration versioning. --- # 159. EDGE RUNTIME Node APIs, local filesystems, and raw TCP sockets may be unavailable. Verify runtime compatibility. --- # 160. KUBERNETES If not present: **NOT APPLICABLE** If present: proceed in detail. --- # 161. REQUESTS / LIMITS CPU and memory allocations. --- # 162. NO MEMORY LIMIT A single runaway pod can exhaust all node memory. --- # 163. LIMIT TOO LOW OOMKill crash loop. --- # 164. CPU LIMIT Throttling can severely degrade latency. --- # 165. HPA Autoscaling metrics must correspond to actual workload behavior. --- # 166. PDB PodDisruptionBudget can safeguard availability during voluntary disruptions. Does not resolve underlying node hardware outages alone. --- # 167. ANTI-AFFINITY If all replicas land on the same node: node failure takes down the entire application. --- # 168. MULTI-ZONE If business requirements demand zone resilience. --- # 169. NODE DRAIN Graceful termination workflows. --- # 170. K8S ROLLING UPDATE `maxUnavailable`, `maxSurge`. --- # 171. READINESS PROBE Critical for safe rolling updates. --- # 172. LIVENESS PROBE Must not trigger cascading restarts across the cluster. --- # 173. INIT CONTAINER Failure can block rollouts indefinitely. --- # 174. JOB / CRONJOB ConcurrencyPolicy, history limits, and execution deadlines. --- # 175. K8S SECRET Base64 encoding is not encryption. --- # 176. SERVICE ACCOUNT RBAC scope minimization. --- # 177. PRIVILEGED CONTAINER Security audit cross-reference. --- # 178. HOST MOUNTS High-risk architectural pattern. --- # 179. INGRESS Routing, TLS termination, and authentication. --- # 180. NETWORK POLICY Defense-in-depth matched to the cluster security model. --- # 181. CONTAINER If using Docker or container runtimes: audit the production image. --- # 182. ROOT USER Running as root escalates container breakout and exploit severity. Severity assessed according to threat exposure. --- # 183. READ-ONLY ROOT FILESYSTEM Hardening where workload compatible. --- # 184. CAPABILITIES Drop unnecessary Linux capabilities. --- # 185. IMAGE SIZE More packages equal larger patch and attack surface, but is not a security finding by itself. --- # 186. DEBUG TOOLS IN IMAGE Shell and curl packages can assist attackers post-compromise, but primary remediation is preventing compromise. --- # 187. MULTI-STAGE BUILD Reduces build tooling and credentials inside runtime images. --- # 188. IMAGE TAG Immutable deployments are more reliable with digest-pinned or strictly versioned artifacts. --- # 189. `latest` Makes it difficult to determine what code is actually deployed. --- # 190. IMAGE VULNERABILITIES Supply chain audit handles in greater depth. --- # 191. LOGGING Audit: - stdout - files - aggregation - retention - redaction --- # 192. LOCAL LOG FILE Can exhaust host disk space. --- # 193. LOG ROTATION If local logs exist. --- # 194. STRUCTURED LOGS Facilitate operational analysis, but are not a mandatory requirement for simple systems. --- # 195. CORRELATION ID Aids tracing across distributed multi-service flows. --- # 196. SECRET REDACTION Security audit cross-reference. --- # 197. LOG LEVEL Production debug logging can increase: - cost - secret exposure - noise --- # 198. LOGGING FAILURE Remote logging infrastructure outages should not automatically crash the application. --- # 199. METRICS Minimum useful dimensions: - throughput - errors - latency - saturation --- # 200. SLO If not defined: **SLO NOT DEFINED** Do not invent. --- # 201. P50/P95/P99 Average latency is insufficient for detecting tail performance degradation. --- # 202. ERROR RATE Distinguish between: - expected 4xx client errors - actual server failures --- # 203. SATURATION CPU is not the sole saturation metric. Track: - memory - DB pool - queue - disk - connections --- # 204. HIGH-CARDINALITY METRICS Using user IDs or unvalidated raw URL query parameters as metric labels can crash monitoring systems. --- # 205. TRACING Valuable for distributed topologies. Do not mandate inside simple monolithic architectures without necessity. --- # 206. TRACE SAMPLING Balance cost against observability fidelity. --- # 207. ALERTING An alert must signify an actionable incident requiring human intervention. --- # 208. ALERT ON CAUSE VS SYMPTOM 80% CPU utilization is not inherently an incident. User-facing errors and SLO error-budget burn are frequently more meaningful. --- # 209. ALERT FATIGUE Excessive noise degrades incident detection. --- # 210. NO ALERT ON CRITICAL FAILURE High-risk condition. Examples: - backup failing for weeks - queue consumer stalled - certificate nearing expiration - DB storage nearly exhausted --- # 211. DISK CAPACITY Audit: - DB - logs - temp - uploads - queue --- # 212. DISK FULL Can trigger: - application crashes - database corruption or failure - inability to write logs --- # 213. DATABASE STORAGE AUTOGROW Managed databases may support autogrowth, but verify: - hard maximum ceiling - cost implications - emergency failure behavior --- # 214. OBJECT STORAGE COST Unbounded upload volume and lack of retention lifecycle policies. --- # 215. BANDWIDTH High-volume downloads and unthrottled uploads. --- # 216. COST AUDIT Do not look solely at the monthly invoice. Identify: - accidental cost amplification - unbounded resource usage - idle assets - excessive overprovisioning --- # 217. AUTOSCALING COST ATTACK An attacker can drive traffic to inflict massive cloud billing spikes prior to triggering outages. --- # 218. PROVIDER COST SMS, email, and external AI APIs. --- # 219. LOGGING COST Excessive production debug logs. --- # 220. EGRESS COST Cross-region transfers and public asset downloads. --- # 221. BUDGET ALERT An operational safeguard, not a technical security mitigation. --- # 222. QUOTAS Cloud and SaaS provider quotas can form hidden hard ceilings. --- # 223. SERVICE QUOTA Ask: > Which quota breaks first under a 10x workload spike? --- # 224. AUTOSCALING Audit: - min - max - metric - cooldown - startup time --- # 225. AUTOSCALING IS NOT INFINITE Constrained by downstream dependency limits. --- # 226. SCALE TO ZERO Can trigger cold-start latency. Acceptable for select background workloads. --- # 227. MAX REPLICA When the ceiling is reached: what happens to incoming traffic? --- # 228. RESOURCE RESERVATION In shared hosting environments: noisy-neighbor risks. --- # 229. CAPACITY HEADROOM Do not mandate an arbitrary percentage. Base headroom on real metrics, SLOs, and historical growth. --- # 230. CHAOS / FAILURE TEST Execute strictly in controlled environments. --- # 231. KILL ONE INSTANCE Does the system maintain continuous availability? --- # 232. DATABASE LATENCY Inject latency during testing. --- # 233. CACHE OUTAGE --- # 234. QUEUE OUTAGE --- # 235. THIRD-PARTY TIMEOUT --- # 236. DNS FAILURE --- # 237. STORAGE FAILURE --- # 238. NETWORK PARTITION If the distributed architecture justifies testing. --- # 239. CLOCK Time drift can compromise: - auth tokens - distributed lease acquisitions - TLS certificates Modern managed infrastructure typically handles NTP. --- # 240. REGION OUTAGE Do not demand multi-region deployment unless business requirements dictate regional failover. However, document the existing blast radius. --- # 241. AVAILABILITY DOMAIN Which single failure domain takes down the entire system? --- # 242. SINGLE POINT OF FAILURE Identify genuine SPOFs: - one VM - one DB - one Redis - one storage account - one external provider --- # 243. SPOF IS NOT AUTOMATICALLY A BUG If downtime tolerance permits it. --- # 244. BUSINESS CRITICALITY Severity must correspond to actual business requirements. --- # 245. DR PLAN Does a documented disaster recovery plan exist? --- # 246. RUNBOOK Critical incidents must have actionable recovery runbooks where operational maturity requires it. --- # 247. BUS FACTOR Operational expertise confined to a single individual is an organizational risk. Report separately from software flaws. --- # 248. ACCESS Who possesses authority to modify production infrastructure? --- # 249. LEAST PRIVILEGE Cloud and administrative access controls. --- # 250. SHARED ADMIN ACCOUNT Degrades accountability and audit attribution. --- # 251. MFA FOR CLOUD ADMIN High-value operational control. --- # 252. BREAK-GLASS If break-glass procedures exist: audit protection and logging mechanisms. --- # 253. PRODUCTION DATABASE ACCESS Developers should generally not require direct write privileges. Dependent upon organizational operational models. --- # 254. AUDIT TRAIL Track infrastructure changes: - deployment - secret change - IAM - DNS - database config --- # 255. FINDING FORMAT Every serious finding must include: ```text ID: Severity: Category: Confidence: Status: Evidence tier: Environment: Component: Provider/platform: Region: Failure domain: Current configuration: Trigger: Failure scenario: T0: T1: T2: T3: User-visible impact: Data impact: Availability impact: Security impact: Recovery impact: Blast radius: Current controls: Why current controls are insufficient: Evidence: Root cause: Recommended remediation: Rollback/recovery considerations: Validation test: Production metric/alert: Complexity: XS / S / M / L / XL ``` --- # 256. SEVERITY Use: ## P0 - CRITICAL - a realistic single failure leads to unrecoverable or catastrophic data loss - infrastructure configuration enables a practical global production compromise - production deployment or recovery mechanisms can systematically destroy critical state without recovery paths - backup and recovery claims are demonstrably false for critical data and a realistic catastrophic scenario exists ## P1 - HIGH - a realistic single point of failure causes major production outages violating defined requirements - staging or preview environment trust permits production compromise - deployment strategies contain a concrete, high-probability outage or data corruption path - critical backups exist but cannot be restored or used - autoscaling or deployments can exhaust shared database connections and collapse the system ## P2 - MEDIUM - significant reliability or operational weakness - limited outage or data loss exposure window - important observability or recovery gap - meaningful environment or configuration drift ## P3 - LOW - limited operational weakness - minor cost or configuration flaw - constrained failure scenario ## P4 - HARDENING - maturity, automation, redundancy, or process improvements without a confirmed current production risk --- # 257. CONFIDENCE Use: ```text HIGH MEDIUM LOW ``` --- # 258. STATUS Use: ```text CONFIRMED LIKELY THEORETICAL NOT VERIFIED ``` --- # 259. EVIDENCE TIER ```text A - reproduced or production telemetry B - complete config/topology/failure path C - strong infrastructure/code evidence D - partial/inferred E - general hardening ``` --- # 260. CATEGORY Use: ```text DEPLOYMENT CONFIGURATION COMPUTE DATABASE CACHE QUEUE STORAGE NETWORK DNS TLS AUTOSCALING BACKUP DISASTER RECOVERY OBSERVABILITY SECURITY COST KUBERNETES SERVERLESS CONTAINER ``` --- # 261. FALSE-POSITIVE PREVENTION Before reporting a P0/P1/P2 finding verify: 1. actual provider and platform 2. deployed environment 3. current configuration 4. topology 5. redundancy 6. provider-managed behaviors 7. backup and recovery mechanisms 8. autoscaling policies 9. business and SLO requirements 10. production telemetry evidence where accessible --- # 262. DO NOT REPORT SINGLE INSTANCE AUTOMATICALLY AS P1 If the application can tolerate brief downtime: a single instance may represent an entirely rational architecture. --- # 263. DO NOT DEMAND MULTI-REGION AUTOMATICALLY Multi-region introduces substantial architectural complexity. --- # 264. DO NOT DEMAND KUBERNETES AUTOMATICALLY PaaS, serverless, or simple VMs may be superior choices. --- # 265. DO NOT DEMAND TERRAFORM AUTOMATICALLY IaC is an instrument, not an ultimate objective. --- # 266. DO NOT DEMAND REDIS AUTOMATICALLY Avoid unless there is a genuine architectural requirement. --- # 267. DO NOT DEMAND READ REPLICAS AUTOMATICALLY Unnecessary in the absence of a confirmed read bottleneck. --- # 268. DO NOT DEMAND BLUE/GREEN FOR EVERY SYSTEM Deployment strategies must align with actual platform capabilities and risk profiles. --- # 269. DO NOT TREAT MANAGED SERVICES AS MAGIC Managed services reduce operational burdens, but tiering and configuration dictate actual reliability. --- # 270. DO NOT TREAT "BACKUP ENABLED" AS PROOF OF RECOVERY A verified restore is the only proof of recovery. --- # 271. DO NOT TREAT "AUTOSCALING ENABLED" AS PROOF OF SCALABILITY Shared downstream bottlenecks remain unmitigated. --- # 272. DO NOT ALTER INFRASTRUCTURE DURING THE AUDIT Without explicit authorization do not: - deploy - scale - restart production services - modify DNS records - change firewall rules - rotate secrets - restore backups - delete resources - alter IAM permissions First complete the audit. --- # 273. OUTPUT - DEVOPS_INFRASTRUCTURE_AUDIT.md Structure the final report: ## 1. Executive Summary - infrastructure model - environments - deployment model - top production risks - recovery posture ## 2. Infrastructure Architecture ## 3. Environment Isolation ## 4. Configuration Management ## 5. Compute / Runtime Audit ## 6. Statelessness / Local State Audit ## 7. Health / Readiness / Startup Audit ## 8. Deployment Strategy Audit ## 9. Migration / Schema Deployment Audit ## 10. Rollback Audit ## 11. Graceful Shutdown Audit ## 12. Database Infrastructure Audit ## 13. Cache Infrastructure Audit ## 14. Queue / Worker Infrastructure Audit ## 15. Storage Audit ## 16. Backup Audit ## 17. Restore Audit ## 18. Disaster Recovery Audit ## 19. External Dependency Resilience ## 20. Network / Firewall Audit ## 21. CDN / Load Balancer Audit ## 22. DNS / Domain Audit ## 23. TLS / Certificate Audit ## 24. Autoscaling Audit ## 25. Serverless Audit If relevant. ## 26. Kubernetes Audit If relevant. ## 27. Container Audit If relevant. ## 28. Logging Audit ## 29. Metrics / Tracing Audit ## 30. Alerting Audit ## 31. Capacity / Quota Audit ## 32. Cost Resilience Audit ## 33. Cloud/IAM Operational Audit ## 34. Failure Injection Findings ## 35. Findings Summary | ID | Severity | Component | Category | Failure | Blast radius | Confidence | |---|---|---|---|---|---|---| ## 36. P0 Findings ## 37. P1 Findings ## 38. P2 Findings ## 39. P3 Findings ## 40. P4 Hardening ## 41. Things Done Well ## 42. Not Applicable ## 43. Not Verified ## 44. Production Remediation Roadmap --- # 274. INFRASTRUCTURE MATRIX | Component | Provider | Region | Redundancy | Persistent | Critical | |---|---|---|---|---|---| --- # 275. ENVIRONMENT MATRIX | Resource | Dev | Preview | Staging | Production | Shared | |---|---|---|---|---|---| --- # 276. FAILURE DOMAIN MATRIX | Component | Failure | User impact | Data impact | Recovery | |---|---|---|---|---| --- # 277. BACKUP MATRIX | Data | Backup | Frequency | Retention | PITR | Restore tested | |---|---|---|---|---|---| --- # 278. DEPLOYMENT MATRIX | Component | Strategy | Mixed version | Rollback | Migration coupling | |---|---|---|---|---| --- # 279. OBSERVABILITY MATRIX | Failure | Metric | Log | Alert | Runbook | |---|---|---|---|---| --- # 280. SECOND PASS - KILL ONE APPLICATION INSTANCE In a controlled environment, evaluate: - user requests - sessions - uploads - jobs - WebSockets --- # 281. SECOND PASS - FULL APPLICATION RESTART Ask what is lost during a complete cold restart. --- # 282. SECOND PASS - DEPLOYMENT AT PEAK LOAD Simulate mentally or via testing: ```text peak traffic + rolling deploy ``` Ask whether remaining instances have adequate headroom. --- # 283. SECOND PASS - FAILED DEPLOY AT 50% Fleet split half new and half old. Verify API, schema, and job compatibility. --- # 284. SECOND PASS - ROLLBACK Revert to the previous artifact in staging or test environments. Verify: - DB - config - jobs - assets --- # 285. SECOND PASS - DATABASE FAILOVER If the platform supports failover drills: verify reconnection and recovery. If not: analyze documented platform behaviors and client configurations. --- # 286. SECOND PASS - DB CONNECTION STORM Simulate application autoscaling: ```text 1 -> 10 -> max ``` Calculate total connection budgets against database pool limits. --- # 287. SECOND PASS - CACHE DOWN Ask: - does the application continue operating? - does the database survive the query storm? - what happens to sessions? - what happens to distributed locks? --- # 288. SECOND PASS - QUEUE DOWN Ask: - do requests fail? - are jobs dropped? - does local memory buffer overflow? --- # 289. SECOND PASS - STORAGE DOWN Ask: - uploads - downloads - app startup - critical user path --- # 290. SECOND PASS - THIRD-PARTY 30s LATENCY Verify: - timeouts - worker thread exhaustion - retries - cascading failures --- # 291. SECOND PASS - CERTIFICATE EXPIRY Determine: - renewal ownership - automation reliability - monitoring alerts --- # 292. SECOND PASS - DOMAIN/DNS CHANGE Ask: - TTL values - rollback procedures - stale cached records - TLS certificate alignment --- # 293. SECOND PASS - DISK 95% For persistent disks and databases: verify whether there is: - alerting - automatic volume expansion - cleanup routines - emergency response protocols --- # 294. SECOND PASS - BACKUP RESTORE The most critical operational validation. In an isolated environment execute: ```text backup ↓ restore ↓ application connect ↓ integrity validation ``` --- # 295. SECOND PASS - ACCIDENTAL TABLE DELETE Ask: > What exact recovery procedure exists? --- # 296. SECOND PASS - BAD MIGRATION Ask: - rollback feasibility - PITR availability - forward fix complexity - required downtime --- # 297. SECOND PASS - STAGING COMPROMISE Ask what production systems staging can reach. --- # 298. SECOND PASS - PROVIDER QUOTA For each managed service identify the hard quota relevant to the workload. --- # 299. SECOND PASS - REGION/AZ FAILURE If the system claims high availability: verify whether replicas and data genuinely cross failure domains. --- # 300. SECOND PASS - OBSERVABILITY OUTAGE If logging and monitoring services fail: does the application continue running? --- # 301. SECOND PASS - LOG STORM Simulate a severe error surge. Ask impact on: - disk capacity - log ingestion limits - cost - application latency --- # 302. SECOND PASS - AUTOSCALING MAX Scale workload to maximum configured replicas. Ask which downstream bottleneck breaks first. --- # 303. SECOND PASS - INSTANCE STARTUP STORM All instances restart concurrently. Track: - DB connections - cache priming - external providers - migrations --- # 304. SECOND PASS - BACKUP CREDENTIAL COMPROMISE Ask whether a single identity can: ```text delete production + delete all backups ``` If yes: recovery blast radius is catastrophic. --- # 305. SECOND PASS - EXPENSIVE ABUSE A single malicious user or client triggers: - autoscaling spikes - SMS/email quota expenditure - egress bandwidth surges - logging saturation Evaluate cost guardrails. --- # 306. FINAL QUALITY GATE Before finalizing the review verify: - actual platform and provider are identified - environments are comprehensively mapped - production and staging isolation is verified - ephemeral and local state is identified - startup and readiness behaviors are evaluated - deployment strategy aligns with actual platform capabilities - mixed-version rollout periods are accounted for - migrations are evaluated alongside rollout and rollback plans - rollback is not reduced to a simple Git revert - graceful shutdown mechanisms are checked - database connection budgets account for autoscaling ceilings - managed database HA is not assumed without tier and configuration proof - cache and queue failure semantics are defined - backups include verified restore validation - RPO and RTO requirements are not fabricated - DNS, TLS, and domain expiration risks are evaluated - direct-origin, CDN, and WAF boundaries are audited - service quotas are analyzed - autoscaling accounts for downstream dependency capacity - logging and monitoring are evaluated beyond mere presence - alerts correlate to genuine failure modes - cost amplification risks are evaluated - single points of failure are prioritized against actual business requirements - Kubernetes, multi-region, and Terraform are not recommended by default - every P0/P1 finding features a concrete production failure path - P4 operational maturity items are clearly segregated from active production flaws --- # FINAL RULE I do not want a report of the type: > Use Kubernetes, autoscaling, Terraform, multi-region, and monitoring. That is not a DevOps & Infrastructure Audit. I am seeking issues such as: ```text production: 10 app instances each: DB pool = 30 ↓ maximum possible connections: 300 DB plan: max 200 ↓ traffic spike causes autoscaling ↓ new instances open more connections ↓ DB rejects connections ↓ autoscaling makes outage worse ``` or: ```text application writes user uploads to: /tmp/uploads ↓ platform filesystem is ephemeral ↓ instance restart/redeploy ↓ metadata remains in database ↓ actual files disappear ↓ permanent user data loss ``` or: ```text every application replica runs: migrate-on-startup ↓ rolling deploy starts 20 replicas ↓ multiple migration attempts race ↓ DDL lock blocks application queries ↓ readiness checks fail ↓ deployment cascade ``` or: ```text backup: enabled daily ↓ no restore test ever performed ↓ disaster occurs ↓ latest backup requires missing encryption key ↓ backup cannot be restored ↓ "backup enabled" provided false confidence ``` or: ```text CDN protects public domain with WAF and rate limiting ↓ backend origin also has public hostname ↓ origin accepts direct traffic ↓ attacker bypasses CDN ↓ WAF and edge rate limits disappear ``` or: ```text production deployment: new code requires new NOT NULL column ↓ migration executes first ↓ old replicas still serve traffic ↓ old code inserts rows without column value ↓ requests fail during rolling deployment ``` or: ```text critical jobs stored only in process memory ↓ deployment sends SIGTERM ↓ process exits ↓ queued in-memory work disappears ↓ users have successful API acknowledgements ↓ business action never occurs ``` or: ```text production DB and backups use same cloud admin identity ↓ credential compromised ↓ attacker deletes production DB ↓ same credential deletes backups ↓ recovery path disappears ``` or: ```text certificate renewal is manual ↓ no expiry alert ↓ certificate expires Saturday night ↓ all HTTPS traffic fails ↓ application code and infrastructure remain healthy ↓ complete public outage ``` These are the DevOps and infrastructure vulnerabilities you need to discover. Think through: - source - artifact - configuration - deployment - runtime - state - network - dependencies - data - failure - detection - recovery For every serious finding you must be able to answer: > Which concrete infrastructure component is the problem? > What event triggers the failure? > Is the failure realistic in the current deployment? > What is the blast radius? > Do we lose only availability or data as well? > How does the system detect the issue? > How does it restore normal state? > Has recovery ever been genuinely tested? > Does the proposed change actually resolve the root cause or merely add infrastructure complexity? If production topology is unconfirmed: **PRODUCTION TOPOLOGY NOT VERIFIED.** If SLO/RPO/RTO are undefined: **REQUIREMENT NOT DEFINED.** If backups exist but restore is unconfirmed: **RECOVERABILITY NOT VERIFIED.** If the issue represents only a maturity improvement without a confirmed production failure path: **P4 - HARDENING.** It is better to discover 10 genuine infrastructure failure paths with precise recovery consequences than to draft 200 generic DevOps recommendations. The goal is to produce a forensically precise Ultimate DevOps & Infrastructure Audit that directly translates into: - deployment hardening - migration safety plan - rollback strategy - backup/restore testing - environment isolation - autoscaling guardrails - infrastructure monitoring - disaster-recovery plan - production reliability roadmap
分类与标签
编程开发、DevOps审计、云基础设施、生产可靠性