测试套件风险覆盖与断言审计
将关键业务约束映射到现有测试,识别弱断言与故障覆盖缺口,按生产风险提出补测方案。
完整提示词
I want you to perform a maximally deep, systematic, evidence-first and production-oriented audit of the complete test suite of the application or repository. Main objective: > Determine whether the existing test suite actually protects critical business invariants, failure modes and production behavior, or only creates an illusion of safety through a large number of tests that do not cover the most important risks. This is not: - counting test files - a coverage percentage audit without understanding risk - insisting on 100% coverage - an assumption that more tests automatically mean better quality - automatic criticism of mocks - automatic criticism of E2E tests because they are slow - automatically favoring unit tests - a recommendation to test every private helper directly - a replacement for production observability - a replacement for a security audit Priority: **critical correctness > data integrity > authorization/security invariants > regression protection > concurrency/reliability > deployment safety > user flows > maintainability > raw coverage percentage** It is better to have 200 tests that prove critical invariants than 5,000 tests that check trivial implementation details. ## 1. REPOSITORY AND TEST STACK DISCOVERY Before any findings, map: - language - framework - test framework - test runners - assertion libraries - mocking libraries - fixtures - factories - test containers - emulators - browsers - device tests - contract tests - integration tests - E2E tests - performance tests - property-based tests - fuzz tests - security tests - CI/local execution - database strategy - external service strategy If there are several test projects, map all of them. ## 2. TEST TAXONOMY Classify the actual tests, not just the folders: ```text unit component integration contract API database UI E2E smoke regression performance security property fuzz chaos ``` ## 3. CRITICAL FLOW INVENTORY Before the coverage analysis, list the most important flows of the system. For each: ```text Flow: Criticality: Main invariant: External dependencies: Persistence: Authorization: Concurrency: Rollback/recovery: Existing tests: ``` ## 4. BUSINESS INVARIANTS Extract the invariants from the production code. Example: ```text order total cannot become negative user cannot read another tenant's record same webhook cannot create duplicate payment inventory cannot be decremented twice ``` ## 5. INVARIANT TO TEST MAPPING For each critical invariant, find evidence that the test suite checks it. ## 6. HAPPY PATH It is not enough. ## 7. NEGATIVE PATH ## 8. BOUNDARY VALUE ## 9. INVALID STATE ## 10. ERROR PATH ## 11. RETRY PATH ## 12. ROLLBACK PATH ## 13. CONCURRENCY PATH ## 14. PERMISSION PATH ## 15. DATA MIGRATION PATH ## 16. EXTERNAL SERVICE FAILURE ## 17. TEST ORACLE Ask: How does the test know that the result is actually correct? ## 18. WEAK ASSERTION Example: ```text expect(response.status).toBe(200) ``` is not enough if the critical business state can be wrong. ## 19. NO ASSERTION A test that only "does not throw an exception". ## 20. ASSERTION ON WRONG LAYER Mock verified, real state not verified. ## 21. SNAPSHOT TEST A snapshot is not automatically good or bad. Check the signal/noise ratio. ## 22. GOLDEN FILE ## 23. MOCKING Map: - what is mocked - why - what behavior is assumed - what contract remains unverified ## 24. OVER-MOCKING Test mirrors implementation and misses integration. ## 25. UNDER-MOCKING Test becomes slow/fragile without additional confidence. ## 26. FAKE Could be better than brittle mock where appropriate. ## 27. REAL DATABASE ## 28. IN-MEMORY DATABASE Can differ materially from production engine. ## 29. TEST CONTAINERS ## 30. SCHEMA MIGRATIONS Tests should use realistic schema state. ## 31. DATABASE TRANSACTION TEST ## 32. ROLLBACK TEST ## 33. LOCK/CONCURRENCY TEST ## 34. EXTERNAL API ## 35. CONTRACT TEST ## 36. PROVIDER SANDBOX ## 37. API MOCK Must not drift from provider. ## 38. CONSUMER-DRIVEN CONTRACT Where appropriate. ## 39. FRONTEND TESTING Audit: - rendering - state transitions - async - error - accessibility - keyboard - forms - navigation ## 40. UI IMPLEMENTATION DETAIL Avoid tests tightly coupled to DOM internals without need. ## 41. BACKEND TESTING Audit: - business logic - persistence - authorization - validation - error semantics - transactions - external calls ## 42. API TESTS Check: - status - body - side effects - authorization - idempotency - error contract ## 43. AUTHENTICATION TESTS ## 44. AUTHORIZATION TESTS Role matrix. ## 45. TENANT ISOLATION TESTS ## 46. IDOR REGRESSION ## 47. FILE UPLOAD TESTS ## 48. RATE LIMIT TESTS ## 49. WEBHOOK TESTS ## 50. BACKGROUND JOB TESTS ## 51. QUEUE TESTS ## 52. SCHEDULER TESTS ## 53. DUPLICATE DELIVERY ## 54. OUT-OF-ORDER DELIVERY ## 55. RETRY ## 56. DEAD LETTER ## 57. MOBILE TESTS If applicable: - lifecycle - process death - permissions - offline - rotation - background - release build ## 58. DESKTOP TESTS ## 59. BROWSER MATRIX ## 60. DEVICE MATRIX ## 61. VERSION MATRIX ## 62. FEATURE FLAGS Test both paths. ## 63. CONFIGURATION ## 64. ENVIRONMENT DIFFERENCE ## 65. RELEASE BUILD Debug-only success is insufficient. ## 66. TEST DATA Realistic? ## 67. EDGE DISTRIBUTION ## 68. UNICODE ## 69. TIMEZONE ## 70. DST ## 71. CLOCK Use controllable clock where relevant. ## 72. RANDOM Seed/reproducibility. ## 73. GENERATED IDS ## 74. SORT ORDER ## 75. PAGINATION ## 76. LARGE DATASET ## 77. EMPTY DATASET ## 78. DUPLICATES ## 79. NULL ## 80. ZERO ## 81. MAXIMUM ## 82. MALFORMED INPUT ## 83. RACE ## 84. TEST ISOLATION One test should not depend on another. ## 85. ORDER DEPENDENCE ## 86. SHARED GLOBAL STATE ## 87. DATABASE CLEANUP ## 88. PORT CONFLICT ## 89. TEMP FILE ## 90. CACHE ## 91. CLOCK LEAK ## 92. TEST PARALLELISM ## 93. PARALLEL SAFETY ## 94. TEST RETRIES Retry can hide flaky or real defect. ## 95. QUARANTINED TEST ## 96. SKIPPED TEST Audit reason and age. ## 97. TODO TEST ## 98. DISABLED SUITE ## 99. CI VS LOCAL Test suite may run differently. ## 100. ENV VAR ## 101. SECRETS ## 102. SERVICE DEPENDENCY ## 103. NETWORK ACCESS Unexpected live calls. ## 104. DETERMINISM ## 105. FLAKINESS For each flaky test, establish its failure rate, the conditions under which it fails and whether the cause is in the test or in the product. A retry is not a fix. ## 106. TEST DURATION ## 107. SLOW TEST Not automatically bad. ## 108. TEST PYRAMID Do not apply dogmatically. ## 109. TEST PORTFOLIO Optimize for risk, not shape. ## 110. COVERAGE Use: - line - branch - function - mutation where available - critical-path coverage ## 111. COVERAGE BLIND SPOT High line coverage can miss business combinations. ## 112. BRANCH COVERAGE ## 113. CONDITION COVERAGE ## 114. MUTATION TESTING Useful for assertion quality, not mandatory everywhere. ## 115. COVERAGE EXCLUSION ## 116. GENERATED CODE ## 117. DEAD CODE ## 118. CRITICAL FILE WITH LOW COVERAGE ## 119. HIGH COVERAGE LOW VALUE ## 120. MISSING TESTS Rank missing tests by production risk, not by coverage percentage. For each one, propose the lowest test layer that proves the invariant. ## 121. REGRESSION HISTORY Use past bugs/incidents. ## 122. BUG TO TEST Every serious fixed bug should normally gain regression protection where feasible. ## 123. INCIDENT TO TEST ## 124. MIGRATION FAILURE HISTORY ## 125. PRODUCTION DATA SHAPE ## 126. PERFORMANCE REGRESSION ## 127. SECURITY REGRESSION ## 128. TEST MAINTAINABILITY ## 129. DUPLICATE TESTS ## 130. COPY/PASTE ## 131. HELPER ABSTRACTION Too much abstraction can hide intent. ## 132. TEST NAME Should state behavior. ## 133. ARRANGE ACT ASSERT Not mandatory formatting, but intent should be clear. ## 134. FIXTURE COMPLEXITY ## 135. FACTORY DEFAULT Can hide required field assumptions. ## 136. TEST READABILITY ## 137. FAILURE MESSAGE ## 138. DEBUGGABILITY ## 139. OBSERVABILITY OF TEST FAILURE Logs/artifacts/screenshots where useful. ## 140. E2E ARTIFACT Video/screenshot/trace. ## 141. TEST SHARDING ## 142. CACHE Build/test caching can hide stale artifacts if misconfigured. ## 143. CI GATE Which tests actually block merge/release? ## 144. OPTIONAL TESTS ## 145. RELEASE GATE ## 146. POST-DEPLOY SMOKE ## 147. MONITORING AS TEST Production monitor is not replacement for pre-release test, but can cover live-only invariants. ## 148. FALSE POSITIVE RULES Do not automatically report: - low global coverage - high global coverage - many mocks - no mocks - slow integration test - absence of E2E - absence of unit test for trivial getter - snapshot tests - skipped test A finding must show a concrete risk or coverage gap. ## 149. EVIDENCE TIERS ```text A - test execution, mutation result, reproduced bug or production incident evidence B - complete code-to-test mapping proving gap/weakness C - strong static evidence D - suspected test weakness requiring verification E - test hardening/maturity ``` ## 150. STATUS MODEL ```text CONFIRMED LIKELY NOT VERIFIED CONTROLLED NOT APPLICABLE HARDENING ``` ## 151. SEVERITY P0: - test suite systematically allows catastrophic corruption/security/release failure with no guard where automated prevention is feasible P1: - critical business/security invariant untested and demonstrably vulnerable to regression - release suite misses known severe production failure path P2: - material coverage or assertion weakness P3: - limited quality/maintainability weakness P4: - test maturity improvement ## 152. FINDING FORMAT ```text ID: Severity: Status: Evidence tier: Component/flow: Invariant: Existing tests: Missing/weak behavior: Failure scenario: Production impact: Evidence: Why current test misses it: Recommended test layer: Concrete test cases: Regression risk: ``` ## 153. MATRICES ### Critical Flow Coverage Matrix | Flow | Happy | Error | Auth | Concurrency | Recovery | |---|---|---|---|---|---| ### Invariant Matrix | Invariant | Unit | Integration | E2E | Production signal | |---|---|---|---|---| ### Test Layer Matrix | Behavior | Current layer | Appropriate layer | Gap | |---|---|---|---| ## 154. SECOND PASS Re-check the suite through: - known past bug - invalid auth - cross-tenant access - duplicate event - timeout - partial failure - process restart - concurrent request - empty data - very large data - release build - feature flag off/on - production-like database - external API contract drift ## 155. FINAL QUALITY GATE Confirm that you have covered: - critical flows - invariants - assertions - mocks/fakes - DB - API - auth - concurrency - external services - jobs/queues - failure/recovery - UI - release configuration - data boundaries - flaky/skipped tests - coverage - regressions - test maintainability - CI/release gating ## 156. OUTPUT `ULTIMATE_TEST_SUITE_AUDIT.md` ## 157. FAILURE CHAINS ```text webhook handler has 95% line coverage ↓ tests call handler only once per event ↓ provider retries same webhook ↓ idempotency path is never tested ↓ same payment is processed twice in production ``` ```text authorization tests mock repository ↓ mock always returns tenant-scoped record ↓ real query misses tenant filter ↓ test suite stays green ↓ cross-tenant data leak reaches production ``` # FINAL RULE Do not judge a test suite by the number of tests. Judge it by the question: > If the most important production invariant breaks tomorrow, is there a test that will reliably stop it before release?
分类与标签
编程开发、测试覆盖分析、断言质量、业务约束、回归测试、发布质量门禁