返回提示词目录第 1360 页

定位不稳定测试的非确定性根因

通过重复运行、调整顺序与并发条件追踪测试波动来源,验证根因并提出避免重试掩盖问题的修复方案。

完整提示词

I want a forensic analysis of flaky tests with the goal of identifying the exact nondeterministic dependency and removing the root cause, instead of masking the problem with retries.

Main objective:

> Find tests whose result depends on timing, order, environment, shared state, randomness, the network or a hidden dependency, and prove the root cause with a reproducible method.

This is not:

- automatically deleting a flaky test
- increasing a timeout without evidence
- `retry: 3` as a fix
- an assumption that a test that failed once must be flaky
- an assumption that a slow test must be flaky

## 1. FLAKE EVIDENCE

Look for:

- repeated CI failures
- rerun pass
- local vs CI difference
- order-dependent pass/fail
- timing sensitivity

## 2. REPRODUCTION

Run repeatedly.

## 3. SEED

## 4. ORDER

Randomize test order.

## 5. PARALLEL

Compare serial vs parallel.

## 6. RESOURCE CONTENTION

## 7. CLOCK

## 8. SLEEP

## 9. FIXED DELAY

## 10. POLLING

## 11. EVENTUAL CONSISTENCY

## 12. ASYNC NOT AWAITED

## 13. BACKGROUND TASK

## 14. TIMER

## 15. UI ANIMATION

## 16. NETWORK

## 17. EXTERNAL API

## 18. DNS

## 19. RATE LIMIT

## 20. PORT

## 21. TEMP FILE

## 22. FILE LOCK

## 23. DATABASE

## 24. TRANSACTION LEAK

## 25. DATA CLEANUP

## 26. UNIQUE CONSTRAINT

## 27. SHARED DB

## 28. TEST FIXTURE COLLISION

## 29. RANDOM ID COLLISION

## 30. CACHE

## 31. GLOBAL SINGLETON

## 32. ENV VAR MUTATION

## 33. LOCALE

## 34. TIMEZONE

## 35. DST

## 36. CURRENT DATE

## 37. RANDOMNESS

## 38. UNSEEDED RANDOM

## 39. HASH/ITERATION ORDER

## 40. THREADING

## 41. RACE

## 42. PROCESS

## 43. CPU LOAD

## 44. MEMORY PRESSURE

## 45. GC

## 46. BROWSER

## 47. DOM READINESS

## 48. SELECTOR

## 49. STALE ELEMENT

## 50. SCREENSHOT

Rendering variability.

## 51. DEVICE

## 52. EMULATOR

## 53. OS VERSION

## 54. CI RUNNER

## 55. CONTAINER STARTUP

## 56. SERVICE HEALTH

## 57. DEPENDENCY VERSION

## 58. TEST ORDER

## 59. SHARED STATIC STATE

## 60. BEFORE/AFTER HOOK

## 61. MISSING CLEANUP

## 62. RETRY HIDING DEFECT

## 63. PRODUCTION RACE

Important: test flake can reveal real product race.

## 64. TEST-ONLY RACE

Differentiate.

## 65. QUARANTINE

Temporary only, with owner/reason.

## 66. DISABLE

Requires justification.

## 67. TIMEOUT INCREASE

Only if expected operation legitimately requires longer bound.

## 68. CONDITION WAIT

Prefer observable condition over arbitrary sleep.

## 69. VIRTUAL CLOCK

Where appropriate.

## 70. DETERMINISTIC DATA

## 71. ISOLATED RESOURCE

## 72. UNIQUE NAMESPACE

## 73. TEST CONTAINER

## 74. FAKE SERVICE

## 75. CONTRACT

## 76. LOGGING

Capture enough to prove race.

## 77. TIMELINE

Build event sequence.

## 78. TRACE

## 79. FAILURE RATE

## 80. CONDITIONAL RATE

By runner, order, time.

## 81. STATISTICAL REPRO

## 82. FALSE POSITIVE RULES

Do not call it flaky:

- deterministic real regression
- consistent platform incompatibility
- persistent infrastructure outage
- intentionally randomized property test that exposes real failure

## 83. EVIDENCE TIERS

```text
A - repeated controlled reproduction of nondeterministic pass/fail
B - exact race/order/timing path proven
C - strong statistical/log evidence
D - suspected source
E - hardening
```

## 84. STATUS

```text
CONFIRMED
LIKELY
NOT VERIFIED
CONTROLLED
NOT APPLICABLE
HARDENING
```

## 85. SEVERITY

P0:
rarely applicable, only if flake hides catastrophic product failure gate

P1:
frequent flake masks/retries critical regression or makes release signal unreliable

P2:
material CI instability

P3:
isolated low-frequency flake

P4:
stability improvement

## 86. FINDING FORMAT

```text
ID:
Severity:
Status:
Evidence tier:
Test:
Failure signature:
Pass rate:
Failure rate:
Environment:
Order:
Parallelism:
Timeline:
Shared dependency:
Root cause:
Product bug or test bug:
Fix:
Proof after fix:
```

## 87. FLAKE MATRIX

| Test | Rate | Serial | Parallel | Seed/order | Root cause |
|---|---:|---|---|---|---|

## 88. SECOND PASS

For suspected flaky test:

- run 100x where practical
- randomize order
- vary seed
- serial
- parallel
- CPU constrained
- clock controlled
- timezone changed
- external dependency disabled
- cleanup verified

## 89. FINAL QUALITY GATE

Confirm:

- nondeterminism proven
- root cause identified or explicitly not verified
- retries not mistaken for fix
- product race considered
- deterministic fix validated
- repeated post-fix runs pass

## 90. OUTPUT

`FLAKY_TEST_HUNTER.md`

## 91. FAILURE CHAINS

```text
test clicks "Save"
↓
asserts immediately
↓
database write completes asynchronously
↓
fast runner passes
↓
loaded CI runner asserts before persistence
↓
intermittent failure
```

```text
tests share same user email
↓
parallel execution
↓
both create user
↓
one gets unique constraint
↓
failure depends on scheduling
```

# FINAL RULE

A flaky test is not fixed when:

```text
it no longer fails often
```

but when the nondeterministic dependency has been removed or controlled and the test result again depends only on the behavior it tests.

分类与标签

编程开发、自动化测试、CI稳定性、测试并发

前往 AI 图片创作