定位不稳定测试的非确定性根因
通过重复运行、调整顺序与并发条件追踪测试波动来源,验证根因并提出避免重试掩盖问题的修复方案。
完整提示词
I want a forensic analysis of flaky tests with the goal of identifying the exact nondeterministic dependency and removing the root cause, instead of masking the problem with retries. Main objective: > Find tests whose result depends on timing, order, environment, shared state, randomness, the network or a hidden dependency, and prove the root cause with a reproducible method. This is not: - automatically deleting a flaky test - increasing a timeout without evidence - `retry: 3` as a fix - an assumption that a test that failed once must be flaky - an assumption that a slow test must be flaky ## 1. FLAKE EVIDENCE Look for: - repeated CI failures - rerun pass - local vs CI difference - order-dependent pass/fail - timing sensitivity ## 2. REPRODUCTION Run repeatedly. ## 3. SEED ## 4. ORDER Randomize test order. ## 5. PARALLEL Compare serial vs parallel. ## 6. RESOURCE CONTENTION ## 7. CLOCK ## 8. SLEEP ## 9. FIXED DELAY ## 10. POLLING ## 11. EVENTUAL CONSISTENCY ## 12. ASYNC NOT AWAITED ## 13. BACKGROUND TASK ## 14. TIMER ## 15. UI ANIMATION ## 16. NETWORK ## 17. EXTERNAL API ## 18. DNS ## 19. RATE LIMIT ## 20. PORT ## 21. TEMP FILE ## 22. FILE LOCK ## 23. DATABASE ## 24. TRANSACTION LEAK ## 25. DATA CLEANUP ## 26. UNIQUE CONSTRAINT ## 27. SHARED DB ## 28. TEST FIXTURE COLLISION ## 29. RANDOM ID COLLISION ## 30. CACHE ## 31. GLOBAL SINGLETON ## 32. ENV VAR MUTATION ## 33. LOCALE ## 34. TIMEZONE ## 35. DST ## 36. CURRENT DATE ## 37. RANDOMNESS ## 38. UNSEEDED RANDOM ## 39. HASH/ITERATION ORDER ## 40. THREADING ## 41. RACE ## 42. PROCESS ## 43. CPU LOAD ## 44. MEMORY PRESSURE ## 45. GC ## 46. BROWSER ## 47. DOM READINESS ## 48. SELECTOR ## 49. STALE ELEMENT ## 50. SCREENSHOT Rendering variability. ## 51. DEVICE ## 52. EMULATOR ## 53. OS VERSION ## 54. CI RUNNER ## 55. CONTAINER STARTUP ## 56. SERVICE HEALTH ## 57. DEPENDENCY VERSION ## 58. TEST ORDER ## 59. SHARED STATIC STATE ## 60. BEFORE/AFTER HOOK ## 61. MISSING CLEANUP ## 62. RETRY HIDING DEFECT ## 63. PRODUCTION RACE Important: test flake can reveal real product race. ## 64. TEST-ONLY RACE Differentiate. ## 65. QUARANTINE Temporary only, with owner/reason. ## 66. DISABLE Requires justification. ## 67. TIMEOUT INCREASE Only if expected operation legitimately requires longer bound. ## 68. CONDITION WAIT Prefer observable condition over arbitrary sleep. ## 69. VIRTUAL CLOCK Where appropriate. ## 70. DETERMINISTIC DATA ## 71. ISOLATED RESOURCE ## 72. UNIQUE NAMESPACE ## 73. TEST CONTAINER ## 74. FAKE SERVICE ## 75. CONTRACT ## 76. LOGGING Capture enough to prove race. ## 77. TIMELINE Build event sequence. ## 78. TRACE ## 79. FAILURE RATE ## 80. CONDITIONAL RATE By runner, order, time. ## 81. STATISTICAL REPRO ## 82. FALSE POSITIVE RULES Do not call it flaky: - deterministic real regression - consistent platform incompatibility - persistent infrastructure outage - intentionally randomized property test that exposes real failure ## 83. EVIDENCE TIERS ```text A - repeated controlled reproduction of nondeterministic pass/fail B - exact race/order/timing path proven C - strong statistical/log evidence D - suspected source E - hardening ``` ## 84. STATUS ```text CONFIRMED LIKELY NOT VERIFIED CONTROLLED NOT APPLICABLE HARDENING ``` ## 85. SEVERITY P0: rarely applicable, only if flake hides catastrophic product failure gate P1: frequent flake masks/retries critical regression or makes release signal unreliable P2: material CI instability P3: isolated low-frequency flake P4: stability improvement ## 86. FINDING FORMAT ```text ID: Severity: Status: Evidence tier: Test: Failure signature: Pass rate: Failure rate: Environment: Order: Parallelism: Timeline: Shared dependency: Root cause: Product bug or test bug: Fix: Proof after fix: ``` ## 87. FLAKE MATRIX | Test | Rate | Serial | Parallel | Seed/order | Root cause | |---|---:|---|---|---|---| ## 88. SECOND PASS For suspected flaky test: - run 100x where practical - randomize order - vary seed - serial - parallel - CPU constrained - clock controlled - timezone changed - external dependency disabled - cleanup verified ## 89. FINAL QUALITY GATE Confirm: - nondeterminism proven - root cause identified or explicitly not verified - retries not mistaken for fix - product race considered - deterministic fix validated - repeated post-fix runs pass ## 90. OUTPUT `FLAKY_TEST_HUNTER.md` ## 91. FAILURE CHAINS ```text test clicks "Save" ↓ asserts immediately ↓ database write completes asynchronously ↓ fast runner passes ↓ loaded CI runner asserts before persistence ↓ intermittent failure ``` ```text tests share same user email ↓ parallel execution ↓ both create user ↓ one gets unique constraint ↓ failure depends on scheduling ``` # FINAL RULE A flaky test is not fixed when: ```text it no longer fails often ``` but when the nondeterministic dependency has been removed or controlled and the test result again depends only on the behavior it tests.
分类与标签
编程开发、自动化测试、CI稳定性、测试并发