返回提示词目录第 1370 页

生产故障模拟与应急演练设计

围绕受控故障场景制定演练方案,检验告警、处置、恢复与沟通流程,并整理证据和改进项。

完整提示词

I want you to design a controlled production-incident simulation / game-day exercise for the application, with the goal of testing detection, triage, containment, recovery, communication and post-incident learning without actually endangering users or data.

Main objective:

> Prove that the team and the system can recognize, understand, contain and recover from a realistic incident under time pressure, while validating the runbooks, observability and recovery assumptions.

This is not:

- causing a real production incident
- destructive chaos without safety boundaries
- a test of people as "culprits"
- a hidden punitive exercise
- a generic tabletop without technical injects

## 1. SIMULATION SCOPE

Define:

- environment
- systems
- participants
- observers
- allowed actions
- forbidden actions
- stop conditions

## 2. SAFETY

Prefer:

- staging
- sandbox
- isolated production-like environment

Production only with explicit safeguards/authorization.

## 3. INCIDENT CLASS

Choose realistic:

- DB unavailable
- DB slow
- queue backlog
- duplicate events
- provider outage
- credential expiry
- bad deployment
- config error
- data inconsistency
- cache corruption
- storage full
- DNS failure
- region failure
- security signal

## 4. BUSINESS IMPACT

## 5. INITIAL SIGNAL

What team sees first?

## 6. ALERT

## 7. USER REPORT

## 8. DASHBOARD

## 9. LOG

## 10. TRACE

## 11. METRIC

## 12. HIDDEN ROOT CAUSE

Participants should diagnose.

## 13. TIMELINE

## 14. INJECT

At controlled times.

## 15. ESCALATION

## 16. SECONDARY FAILURE

## 17. MISLEADING SIGNAL

Use sparingly.

## 18. TRIAGE

## 19. INCIDENT COMMAND

## 20. OWNER

## 21. COMMUNICATION

## 22. STATUS UPDATE

## 23. CUSTOMER IMPACT

## 24. CONTAINMENT

## 25. FEATURE DISABLE

## 26. TRAFFIC SHED

## 27. ROLLBACK

## 28. FAILOVER

## 29. RESTORE

## 30. RESTART

## 31. SECRET ROTATE

## 32. QUEUE DRAIN

## 33. DATA RECONCILE

## 34. RECOVERY VERIFY

## 35. FALSE RECOVERY

System looks healthy but invariant still broken.

## 36. DATA INTEGRITY CHECK

## 37. BUSINESS TRANSACTION CHECK

## 38. SYNTHETIC CHECK

## 39. RPO

## 40. RTO

## 41. ACTUAL RECOVERY TIME

## 42. RUNBOOK

Did it work?

## 43. MISSING STEP

## 44. STALE COMMAND

## 45. PERMISSION

Does responder have access?

## 46. MFA/ACCOUNT

## 47. CREDENTIAL

## 48. TOOL AVAILABILITY

## 49. DEPENDENCY STATUS

## 50. EXTERNAL CONTACT

## 51. DECISION LOG

## 52. TIMESTAMP

## 53. HANDOFF

## 54. FATIGUE

## 55. SHIFT

## 56. POSTMORTEM

## 57. BLAMELESS

Focus on system.

## 58. ROOT CAUSE

## 59. CONTRIBUTING FACTORS

## 60. DETECTION GAP

## 61. RUNBOOK GAP

## 62. OBSERVABILITY GAP

## 63. TEST GAP

## 64. ARCHITECTURE GAP

## 65. FOLLOW-UP

## 66. OWNER

## 67. DEADLINE

## 68. REGRESSION TEST

## 69. GAME DAY REPEAT

## 70. METRICS

Measure:

- detection time
- acknowledgment
- diagnosis
- containment
- recovery
- verification

## 71. FALSE POSITIVE RULES

Do not mark an "incident response failure" only because the team did not know the root cause in advance.

The exercise tests the process, not memorization of the scenario.

## 72. EVIDENCE TIERS

```text
A - observed during simulation
B - directly verified runbook/tool behavior
C - strong inferred gap
D - hypothesis
E - hardening opportunity
```

A gap is confirmed only with evidence tier A or B (observed during the simulation, or directly verified in the runbook or tool). A tier C or D gap stays NOT VERIFIED until a follow-up exercise or check confirms it; record the evidence tier in every finding.

## 73. SEVERITY

For findings of the simulation:

P0:
in a real incident, the gap could cause a catastrophic unrecoverable outcome

P1:
critical response/recovery gap

P2:
material delay/risk

P3:
limited process weakness

P4:
maturity improvement

## 74. FINDING FORMAT

```text
ID:
Severity:
Status:
Evidence tier:
Simulation step:
Expected response:
Observed response:
Detection:
Decision:
Tool/runbook:
Delay:
Impact if real:
Root cause:
Improvement:
Owner:
Validation:
```

## 75. SIMULATION PLAN FORMAT

```text
Scenario:
Objective:
Environment:
Safety boundaries:
Participants:
Initial inject:
Timeline:
Expected signals:
Escalation injects:
Stop conditions:
Recovery target:
Success criteria:
Observers:
```

## 76. SECOND PASS

Add one unexpected but safe inject:

- provider remains slow after rollback
- queue contains duplicates
- monitoring dashboard unavailable
- primary responder lacks permission
- backup restore completes but data inconsistent
- rollback code incompatible with migrated schema

## 77. FINAL QUALITY GATE

Confirm:

- safety
- realism
- detection
- triage
- containment
- recovery
- data verification
- communication
- runbooks
- permissions
- metrics
- postmortem
- follow-up tests

## 78. OUTPUT

`PRODUCTION_INCIDENT_SIMULATION.md`

## 79. EXAMPLE SCENARIO

```text
10:00
database latency increases 20x
↓
API p95 rises
↓
workers accumulate
↓
queue depth rises
↓
responders see generic timeout alert
↓
at 10:10 external provider is also slowed artificially
↓
team must determine primary vs secondary symptom
↓
containment requires traffic reduction
↓
recovery only counts when queue drains and business transactions reconcile
```

# FINAL RULE

An incident simulation is not successful because the team "solved the incident".

It is successful if, after the exercise, you know:

```text
what the system sees
what the team sees
what it does not see
what it can fix
what it cannot
and how the next incident becomes less risky
```

分类与标签

编程开发、故障演练、应急响应、系统可靠性、故障复盘

前往 AI 图片创作