返回提示词目录第 1372 页

应用故障模式与恢复能力审计

分析依赖超时、重复投递和进程中断等故障,输出恢复路径、风险矩阵及故障注入验证方案。

完整提示词

# RELIABILITY AND FAILURE MODE AUDIT

I want a complete reliability audit of the application from a failure-mode, recovery and degraded-operation perspective.

Main objective:

> Determine how the system reacts when a dependency, process, storage, network, queue, cache or provider works slowly, partially, in duplicate or not at all, and whether the system keeps its critical invariants without silent corruption.

This is not:

- "add retries everywhere"
- a checklist of high-availability buzzwords
- an assumption that a health endpoint returning 200 means a healthy system
- an automatic recommendation of a multi-region architecture
- insisting on 99.999%

## 1. RELIABILITY REQUIREMENTS

For each critical flow, define:

- availability
- correctness
- durability
- recovery time
- recovery point
- degraded behavior

## 2. DEPENDENCY MAP

```text
component
↓
dependency
↓
failure effect
↓
fallback
```

## 3. PROCESS CRASH

## 4. HOST CRASH

## 5. CONTAINER RESTART

## 6. NETWORK PARTITION

## 7. LATENCY

## 8. PACKET LOSS

## 9. DNS

## 10. TLS

## 11. DATABASE DOWN

## 12. DATABASE SLOW

## 13. REPLICA LAG

## 14. CONNECTION EXHAUSTION

## 15. DISK FULL

## 16. STORAGE READ-ONLY

## 17. CACHE DOWN

## 18. CACHE STALE

## 19. QUEUE DOWN

## 20. QUEUE BACKLOG

## 21. DUPLICATE DELIVERY

## 22. OUT-OF-ORDER

## 23. DEAD LETTER

## 24. PROVIDER 5XX

## 25. PROVIDER 429

## 26. PROVIDER TIMEOUT

## 27. PARTIAL SUCCESS

## 28. UNKNOWN OUTCOME

## 29. RETRY

## 30. BACKOFF

## 31. JITTER

## 32. RETRY STORM

## 33. THUNDERING HERD

## 34. CIRCUIT BREAKER

## 35. BULKHEAD

## 36. LOAD SHEDDING

## 37. BACKPRESSURE

## 38. RATE LIMIT

## 39. DEADLINE

## 40. TIMEOUT HIERARCHY

## 41. CANCELLATION

## 42. IDEMPOTENCY

## 43. RECONCILIATION

## 44. COMPENSATION

## 45. CHECKPOINT

## 46. RESUME

## 47. ORPHAN STATE

## 48. LEASE

## 49. LOCK

## 50. FENCING

## 51. CLOCK SKEW

## 52. REGION

## 53. AZ

## 54. QUOTA

## 55. CAPACITY

## 56. AUTOSCALING

## 57. COLD START

## 58. WARMUP

## 59. DEPLOYMENT

## 60. ROLLBACK

## 61. MIGRATION

## 62. OLD/NEW VERSION

## 63. FEATURE FLAG

## 64. CONFIG

## 65. CERTIFICATE EXPIRY

## 66. SECRET ROTATION

## 67. BACKUP

## 68. RESTORE

## 69. DR

## 70. RPO

## 71. RTO

## 72. DATA CORRUPTION

## 73. LOGICAL CORRUPTION

## 74. REPLICATION

## 75. SPLIT BRAIN

Where relevant.

## 76. OBSERVABILITY

## 77. SLI

## 78. SLO

## 79. ERROR BUDGET

Only if useful.

## 80. ALERT

## 81. ALERT FATIGUE

## 82. HEALTH CHECK

## 83. READINESS

## 84. LIVENESS

## 85. DEPENDENCY HEALTH

## 86. BUSINESS HEALTH

## 87. SYNTHETIC

## 88. CHAOS

## 89. FAULT INJECTION

## 90. SAFE ENVIRONMENT

## 91. GAME DAY

## 92. RUNBOOK

## 93. ON-CALL

## 94. INCIDENT COMMUNICATION

## 95. RECOVERY VERIFY

## 96. DATA RECONCILIATION

## 97. FALSE POSITIVE RULES

Do not automatically report:

- single region
- no circuit breaker
- retries
- no multi-region
- synchronous architecture

without a requirement and a concrete failure impact.

## 98. EVIDENCE TIERS

```text
A - fault injection, incident or production evidence
B - complete failure/recovery path
C - strong static evidence
D - plausible failure needing validation
E - resilience hardening
```

## 99. STATUS

```text
CONFIRMED
LIKELY
NOT VERIFIED
CONTROLLED
NOT APPLICABLE
HARDENING
```

## 100. SEVERITY

P0:
catastrophic data loss/corruption or global unrecoverable outage

P1:
critical realistic failure with severe outage/corruption and weak recovery

P2:
material reliability weakness

P3:
limited degradation

P4:
hardening

## 101. FINDING FORMAT

```text
ID:
Severity:
Status:
Evidence tier:
Flow:
Dependency:
Failure:
Detection:
Immediate behavior:
Retry:
Persistent state:
Recovery:
Data risk:
User impact:
Blast radius:
Evidence:
Fix:
Fault-injection test:
Runbook:
```

## 102. FAILURE MODE MATRIX

| Dependency | Failure | Detection | System response | Recovery |
|---|---|---|---|---|

## 103. FLOW RELIABILITY MATRIX

| Flow | Dependency outage | Duplicate | Timeout | Restart | Recovery |
|---|---|---|---|---|---|

## 104. SECOND PASS

Inject:

- DB latency
- DB restart
- queue duplicate
- queue delay
- provider timeout
- disk full
- cache loss
- network partition
- process kill
- deployment rollback
- expired credential
- 10x load
- partial external success

## 105. FINAL QUALITY GATE

Confirm:

- dependencies
- timeouts
- retries
- idempotency
- backpressure
- process restart
- persistence
- queues
- DB
- cache
- providers
- deployment
- config/secrets
- backup/restore
- observability
- fault injection
- runbooks

## 106. OUTPUT

`RELIABILITY_FAILURE_MODE_AUDIT.md`

## 107. FAILURE CHAIN

```text
provider latency increases
↓
application timeout is longer than request deadline
↓
requests accumulate
↓
connection pool saturates
↓
healthy endpoints cannot acquire connections
↓
partial provider slowdown becomes full application outage
```

# FINAL RULE

A reliable system is not a system in which dependencies never fail.

A reliable system is one that:

```text
expects failure
limits the blast radius
preserves invariants
and has a provable recovery path
```

分类与标签

编程开发、系统可靠性、故障注入、灾难恢复、重试幂等性

前往 AI 图片创作