后端性能瓶颈审计与优化验证
结合生产指标、查询计划与负载证据定位后端瓶颈,形成分级审计报告和优化验证方案。
完整提示词
I want you to perform an ultra-deep, systematic, evidence-first, and production-oriented performance analysis of the complete backend system. Main goal: > Identify real latency, throughput, CPU, memory, database, network, serialization, queue, and concurrency bottlenecks capable of degrading response times, diminishing system capacity, escalating infrastructure costs, or causing cascading failures under realistic production load. This is not: - a generic "optimize backend" checklist - automatic caching introduction - automatic Redis integration - automatic migration to microservices - arbitrary concurrency expansion - premature indexing of every database column - focusing solely on average response times - drawing conclusions from isolated local micro-benchmarks - micro-optimizing code outside hot execution paths Focus is on: ```text request ↓ routing/middleware ↓ business logic ↓ database ↓ external dependencies ↓ serialization ↓ network response ``` and the underlying resource model: ```text CPU memory database connections threads/event loop file descriptors network sockets queue workers external API quotas ``` Priority: **critical latency > throughput collapse > resource exhaustion > database bottlenecks > external dependency latency > queue backlog > CPU/memory hotspots > micro-optimizations** Primary principle: > Do not optimize what has not been proven to be a bottleneck. --- # 1. DETERMINE BACKEND STACK Before findings, establish: - language - runtime - framework - async model - deployment - database - ORM - cache - message queue - object storage - reverse proxy - API gateway - CDN - external APIs - background workers - observability tooling --- # 2. DETERMINE PRODUCTION TOPOLOGY Map: ```text client ↓ CDN ↓ load balancer ↓ backend instances ↓ database/cache/queue ``` If unknown: **PRODUCTION TOPOLOGY: NOT VERIFIED** Do not draw scalability conclusions assuming 1 or 50 instances without evidence. --- # 3. DETERMINE PERFORMANCE TARGETS Search for existing: - SLO - SLA - latency budget - throughput target - concurrency target - cost target If absent: do not invent universal arbitrary thresholds. State: **PERFORMANCE BUDGET: NOT DEFINED** --- # 4. AVERAGES ARE NOT ENOUGH Where telemetry exists evaluate: - P50 - P90 - P95 - P99 - max Averages obscure severely degraded tail latency. --- # 5. RPS IS NOT THE ONLY METRIC Also map: - concurrent requests - active DB connections - queue depth - worker utilization - CPU - memory - I/O - upstream latency --- # 6. CRITICAL ENDPOINTS Identify high-priority endpoints by: - traffic volume - business criticality - latency sensitivity - computational cost Do not spend the majority of the audit on rarely invoked admin routes when checkout, login, or home queries represent primary load. --- # 7. HOT PATH MAP For each critical endpoint build: ```text HTTP request ↓ middleware ↓ auth ↓ service ↓ DB query ↓ external API ↓ serialization ↓ response ``` For each step determine: - sync vs async - resource usage - latency - frequency - fan-out --- # 8. LATENCY BUDGET Where traces or metrics exist: break down total execution duration. Example: ```text Total: 600 ms DB: 350 ms external API: 180 ms serialization: 20 ms other: 50 ms ``` Do not optimize a 20 ms step while database access consumes 350 ms. --- # 9. EVIDENCE HIERARCHY Use: ## A - Production traces/metrics highest signal. ## B - Production-like load test ## C - Query plans/profiler ## D - Reproducible benchmark ## E - Static code hotspot ## F - Theoretical suspicion Align severity and confidence strictly with evidence tier. --- # 10. MEASURED VS INFERRED For each performance finding designate: ```text MEASURED CODE-LEVEL BOTTLENECK POTENTIAL RISK NOT MEASURED ``` --- # 11. REQUEST CONCURRENCY Map how many concurrent requests an instance sustains prior to: - CPU saturation - memory ballooning - DB connection pool starvation - thread pool exhaustion - event loop blockage --- # 12. EVENT LOOP In event-loop architectures identify blocking operations: - synchronous filesystem I/O - CPU-heavy loops - synchronous cryptography - blocking subprocess calls - synchronous compression --- # 13. EVENT LOOP BLOCKING Scenario: ```text one request ↓ CPU-heavy sync operation ↓ event loop blocked ↓ unrelated requests wait ``` This drastically elevates tail latency across unrelated endpoints. --- # 14. THREAD-BASED SERVER When frameworks employ thread-per-request or thread pooling: analyze: - thread blocking duration - pool capacity - starvation risks Do not blindly apply Node.js event-loop assumptions to Java or .NET platforms without adjustment. --- # 15. ASYNC DOES NOT MEAN FREE Async I/O reduces idle thread blocking, but: - database engines - CPU cores - system memory - downstream dependencies still impose hard physical limits. --- # 16. CPU PROFILING Where profilers exist: pinpoint: - hot functions - serialization and deserialization - parsing logic - cryptographic routines - compression tasks - regex evaluations - template rendering - computational business loops Never guess CPU hotspots without profiling evidence. --- # 17. CPU-BOUND ENDPOINT If an endpoint performs intensive CPU calculations: ask: > Should the HTTP request thread or event loop execute this work synchronously? --- # 18. OFFLOAD CPU-heavy workloads may belong in: - worker thread pools - background queues - dedicated processing services only if latency and business semantics permit. --- # 19. WORKER POOL SATURATION If all CPU tasks route to a shared thread pool: verify that one feature cannot starve all other jobs. --- # 20. UNBOUNDED PARALLELISM Look for: ```text Promise.all(10000 items) ``` or platform equivalents. --- # 21. PARALLEL != FASTER Dispatching 1,000 parallel operations against the same database or API: often degrades throughput and destabilizes dependencies. --- # 22. CONCURRENCY LIMIT Recommend concurrency bounding only with specific downstream resource limits in mind. --- # 23. DATABASE PERFORMANCE For critical requests catalog all issued queries: | Query | Endpoint | Frequency | Rows | Index | Measured | |---|---|---|---|---|---| --- # 24. SLOW QUERY Where database performance insights or slow query logs exist: rank by: - total execution time - mean latency - execution count - P95/P99 where accessible --- # 25. TOTAL DB TIME A 5 ms query invoked 1,000 times per request is often far worse than a single 300 ms query. --- # 26. N+1 Classic flaw: ```text load 100 orders ↓ query customer for each order ``` --- # 27. ORM HIDDEN N+1 Lazy relation loading can conceal high query counts. Inspect actual executed SQL statements. --- # 28. OVERFETCHING Queries loading: - large text columns - binary BLOBs - deep relation graphs unused by the endpoint. --- # 29. UNDERFETCHING Conversely: executing dozens of tiny queries can be much more expensive than a single well-structured JOIN or batch query. --- # 30. INDEX For critical queries inspect: ```text EXPLAIN EXPLAIN ANALYZE ``` safely in pre-production or sandbox environments. Never invent execution plans. --- # 31. FULL SCAN Full table scans are not inherently defects. A scan across 20 rows is frequently faster than an index lookup. --- # 32. COMPOSITE INDEX Verify column ordering against query predicate filters and sorting clauses. --- # 33. INDEX OVERLOAD Excessive indexes inflate: - write latency - storage consumption - index maintenance overhead --- # 34. SORT Sorting large datasets without supporting indexes is computationally expensive. --- # 35. GROUP BY / AGGREGATION Reporting endpoints running expensive aggregations live on every request. Evaluate frequency and dataset dimensions. --- # 36. COUNT Executing `COUNT(*)` over massive filtered datasets can create severe bottlenecks in pagination APIs. Verify against database metrics before assuming. --- # 37. EXACT TOTAL Determine if the UI genuinely requires an exact total count on every page request. If not, expensive calculations may be eliminated. --- # 38. PAGINATION Unbounded collection endpoints represent major operational risks. --- # 39. OFFSET Large `OFFSET` values require scanning and discarding millions of records. --- # 40. KEYSET Keyset pagination excels on large datasets with stable sorting. Do not recommend for trivial table sizes. --- # 41. QUERY DUPLICATION The same request may execute identical SQL queries multiple times across different architecture layers. --- # 42. REQUEST-SCOPED MEMOIZATION Legitimate only when verified repeated queries exist within the single request cycle. --- # 43. CONNECTION POOL Establish: - min and max pool sizing - acquisition timeout - instance count - database max connection headroom --- # 44. POOL SATURATION If requests wait longer to acquire a connection than queries take to execute: the pool is undersized or downstream queries are dragging. --- # 45. POOL TOO LARGE Enlarging connection pools is not an automatic fix. It can crush database engine memory and CPU. --- # 46. MULTI-INSTANCE POOL Calculate aggregate connections: ```text instances × max pool ``` and compare against database server limits. --- # 47. SERVERLESS CONNECTION STORM Rapid function autoscaling can instantly exhaust database connections. --- # 48. CONNECTION PROXY Factor in pooling proxies like PgBouncer or AWS RDS Proxy if deployed. --- # 49. LONG TRANSACTION Prolonged transactions: - hold row and table locks - increase lock contention - block MVCC vacuuming and garbage collection --- # 50. NETWORK INSIDE TRANSACTION High-signal defect: making external HTTP calls or queue operations while holding database transactions open. --- # 51. LOCK CONTENTION Look for hot rows or contention bottlenecks: Example: ```text all requests update same counter row ``` --- # 52. HOT ROW A database table may scale well overall while a single centralized row acts as a serialization funnel. --- # 53. SEQUENCE / COUNTER Global sequential IDs or counters can become contention chokepoints. --- # 54. DATABASE WRITE AMPLIFICATION A single logical mutation triggering: - numerous secondary indexes - triggers - audit log writes - cascading updates --- # 55. TRIGGERS If database triggers execute, incorporate their overhead into write latency analyses. --- # 56. CACHE Before recommending caching ask: > What specific expensive read operation is repeatedly executed? --- # 57. CACHE HIT RATE If unmeasured: **CACHE HIT RATE: NOT MEASURED** --- # 58. LOW HIT RATE A cache with a 5% hit rate adds architectural complexity with almost zero performance gain. --- # 59. CACHE MISS COST Cache misses add round-trip overhead: ```text cache lookup ↓ DB ↓ cache write ``` --- # 60. CACHE KEY Granular cache keying governs both correctness and hit rates. --- # 61. CACHE STAMPEDE When a popular key expires: ```text 1000 requests ↓ 1000 DB queries ``` simultaneously hitting the database. --- # 62. REQUEST COALESCING Consider singleflight coalescing for hot cache misses. Do not implement without proof of stampedes. --- # 63. TTL Excessively short TTLs destroy hit rates. Excessively long TTLs cause stale data issues. Never prioritize performance over business correctness. --- # 64. MEMORY CACHE In-process local memory caches do not share state across replicas. This can be entirely acceptable depending on access patterns. --- # 65. REDIS Do not introduce Redis simply because the backend needs to "be faster." --- # 66. CDN Static or public cacheable responses can be offloaded to edge CDNs. Personalized or authenticated data demands careful cache headers. --- # 67. SERIALIZATION Massive response payloads consume heavy CPU and serialization cycles. --- # 68. JSON SERIALIZATION Evaluate: - response payload dimensions - deep object nesting - duplicated data structures - serializer framework efficiency --- # 69. LARGE JSON Example: ```text 50 MB JSON ``` is not merely a network transmission problem. It exhausts: - database I/O - application memory - serialization CPU - garbage collection pauses --- # 70. RESPONSE COMPRESSION Compression conserves bandwidth at the expense of CPU cycles. Determine where compression takes place: - application layer - reverse proxy - CDN Never compress redundantly across layers. --- # 71. COMPRESSION LEVEL Maximum compression levels rarely provide the optimal throughput tradeoff. --- # 72. BINARY FORMATS Do not migrate REST JSON to Protocol Buffers for theoretical speed without demonstrating an actual serialization bottleneck. --- # 73. RESPONSE SIZE Measure: - average - P95 - max payload sizes where metrics exist. --- # 74. REQUEST SIZE Parsing massive incoming request bodies can also act as CPU and memory hotspots. --- # 75. STREAMING For large responses and exports: verify whether the backend buffers the complete dataset in RAM before dispatching. --- # 76. FILE DOWNLOAD Favor direct streaming or presigned object storage URLs where architecture permits. --- # 77. FILE UPLOAD Large file uploads must not buffer entirely in system RAM. --- # 78. MULTIPART MEMORY Verify framework defaults for multipart memory thresholds. --- # 79. TEMP DISK Spilling uploads to disk reduces memory consumption, but introduces disk I/O and capacity bottlenecks. --- # 80. OBJECT STORAGE PROXY If backend instances merely proxy multi-gigabyte files from S3/GCS: evaluate whether direct presigned client uploads/downloads make more architectural sense. --- # 81. EXTERNAL API LATENCY For every critical downstream dependency: construct: | Dependency | Calls/request | Latency | Timeout | Parallel | |---|---:|---:|---:|---| --- # 82. SEQUENTIAL CALLS Scenario: ```text A 200 ms ↓ B 300 ms ↓ C 400 ms ``` total minimum duration is ~900 ms. If independent, evaluate parallel execution. --- # 83. PARALLEL CALLS Remember parallelization increases concurrent socket usage and downstream load. --- # 84. FAN-OUT An endpoint invoking 50 downstream services has an inherent tail latency vulnerability. --- # 85. TAIL AMPLIFICATION In wide fan-out calls, request duration is bounded by the slowest individual dependency. --- # 86. BATCH API Where downstreams support batching, consolidate individual round trips. --- # 87. CONNECTION REUSE HTTP clients must utilize connection pooling and HTTP keep-alive. --- # 88. NEW CLIENT PER REQUEST Instantiating fresh HTTP clients or connection pools per incoming request is a severe performance defect. --- # 89. DNS Do not optimize DNS without metrics. However, repeated fresh connections multiply DNS resolution and TLS handshake overhead. --- # 90. TLS HANDSHAKE Connection pooling eliminates repetitive cryptographic TLS negotiation. --- # 91. TIMEOUT Overly generous timeouts tie up server threads and connections during downstream outages. --- # 92. RETRIES Retries directly multiply downstream system load. --- # 93. RETRY STORM Downstream latency spikes: ```text requests timeout ↓ all retry ↓ dependency receives even more load ↓ becomes slower ``` --- # 94. NESTED RETRIES Specifically analyze multiplicative retry behavior across layered services. --- # 95. CIRCUIT BREAKER Circuit breakers protect server capacity during persistent downstream outages. Do not introduce without verified dependency failure modes. --- # 96. QUEUE PERFORMANCE Map: - enqueue throughput - processing throughput - worker concurrency - backlog accumulation - job execution latency --- # 97. LITTLE'S LAW MENTAL MODEL If arrival rates continuously outpace consumption capacity: queue depth grows indefinitely. --- # 98. QUEUE DEPTH If unmeasured: **QUEUE DEPTH: NOT MEASURED** --- # 99. OLDEST JOB AGE Oldest message age is often a more actionable operational metric than raw queue depth. --- # 100. WORKER CONCURRENCY Increasing worker processes is not an automatic solution. Downstream systems or databases can become the true limiting factor. --- # 101. JOB BATCHING Processing tiny individual messages introduces heavy message acknowledgment and scheduling overhead. --- # 102. HUGE JOB A single monolithic 1-hour job eliminates failure isolation. Break down into bounded batches. --- # 103. FAIRNESS Heavy batch processing must not starve latency-sensitive jobs sharing the same worker pool. --- # 104. PRIORITY QUEUE Appropriate only when business domains define explicit priority tiers. --- # 105. CRON PERFORMANCE Periodic jobs can trigger sharp resource spikes on the hour. --- # 106. THUNDERING HERD CRON If all backend replicas execute the same scheduled cron simultaneously: system load multiplies violently. --- # 107. SCHEDULING SPREAD Apply randomized scheduling jitter to smooth recurring traffic spikes. --- # 108. MEMORY Inspect memory allocation profiles: - request payload buffering - internal caches - oversized arrays - retained references - queue payloads - in-memory response construction --- # 109. MEMORY LEAK A memory leak exists when resident set size (RSS) or heap memory climbs steadily without returning to baseline under steady-state load. --- # 110. HIGH MEMORY IS NOT A LEAK High memory consumption due to caches or runtime pre-allocations is not inherently a leak. Growth over time requires verification. --- # 111. OOM Analyze worst-case scenarios for memory consumption across large inputs and batch jobs. --- # 112. LARGE ARRAY Pattern: ```text SELECT millions rows ↓ map() ↓ JSON.stringify() ``` allocates multiple simultaneous copies of the entire dataset in RAM. --- # 113. STREAM / ITERATOR Adopt streaming iterators for massive datasets. Avoid overcomplicating small response payloads. --- # 114. GC Extreme object allocation rates induce severe garbage collection pauses. --- # 115. OBJECT CHURN Do not micro-optimize standard object instantiations without profiler confirmation. --- # 116. BUFFER COPYING Large file and data pipelines can perform redundant buffer allocations and memory copies. --- # 117. CACHE MEMORY LIMIT In-memory caches must enforce explicit size bounds when key spaces are unbounded. --- # 118. USER-SCOPED CACHE LEAK Creating in-memory cache entries per user without eviction policies leaks memory proportionally to total user base. --- # 119. LOG MEMORY Asynchronous logging queues can expand indefinitely if disk or network sinks bottleneck. --- # 120. LOGGING PERFORMANCE Excessive debug logging on high-throughput hot paths introduces significant latency. --- # 121. SYNCHRONOUS LOGGING Synchronous disk or network log writes directly inside request hot paths degrade latency. --- # 122. HUGE LOG PAYLOAD Serializing complete request and response bodies consumes substantial CPU and disk I/O. --- # 123. METRICS OVERHEAD Unbounded high-cardinality metric labels overwhelm telemetry systems. --- # 124. TRACING 100% trace sampling across high-throughput services carries noticeable overhead. Tune sampling rates appropriately. --- # 125. MIDDLEWARE Requests may traverse dozens of middleware layers. Never optimize lightweight functions without telemetry evidence. --- # 126. AUTH Token verification can become a CPU hotspot under extreme throughput: - asymmetric signature verification - remote session verification - caching strategies --- # 127. AUTH DB LOOKUP Querying the database on every request to fetch user sessions is costly. Verify whether this is an intentional business requirement. --- # 128. JWT Stateless token verification removes database lookups, but alters revocation semantics. Never recommend JWTs solely for speed. --- # 129. PERMISSION CHECK N+1 permission lookups represent clear performance bugs. --- # 130. TENANT CONFIG If tenant configuration is queried from the DB on every request: check query frequency and caching feasibility. --- # 131. FEATURE FLAGS Remote HTTP feature flag evaluations in request hot paths introduce unnecessary latency. --- # 132. TEMPLATE RENDERING Server-side rendering, email formatting, and document generation can be CPU-intensive. --- # 133. PDF GENERATION Heavy CPU and memory consumer. Offload to background workers unless immediate synchronous delivery is required. --- # 134. IMAGE PROCESSING Same applies to image resizing and transformation. --- # 135. VIDEO Video transcoding must never execute inside web request handlers. --- # 136. CRYPTO Password hashing algorithms (Argon2, bcrypt) are deliberately designed to be computationally expensive. Never optimize them in ways that compromise security. --- # 137. PASSWORD HASH COST A performance audit must never recommend reducing cryptographic cost parameters to lower latency. --- # 138. ENCRYPTION Bulk data encryption can be CPU-bound, but security compliance takes precedence. --- # 139. REGEX Catastrophic regular expression backtracking is both a performance issue and a ReDoS security vulnerability. Audit only on user-controlled inputs with vulnerable regex patterns. --- # 140. STRING PROCESSING Unbounded parsing and string formatting inside inner loops can emerge as hotspots. --- # 141. DATA TRANSFORMATION Repeated translation across layers: ```text DB row ↓ ORM entity ↓ domain ↓ DTO ↓ JSON intermediate ``` accumulates overhead on massive datasets. Do not dismantle clean architectural layering for micro-optimizations without measurements. --- # 142. API GATEWAY If an API gateway handles: - authentication - request transformations - compression - rate limiting include gateway latency in the overall latency budget. --- # 143. REVERSE PROXY BUFFERING Reverse proxy buffering settings can break streaming response flows. --- # 144. CDN MISS Origin server latency is exposed on CDN cache misses. Separate: - cache hit latency - cache miss latency --- # 145. SERVERLESS COLD START In serverless architectures: measure cold start latency separately from warm execution. --- # 146. BUNDLE/INIT SIZE Bloated dependency trees inflate cold start durations. --- # 147. EAGER INIT Eagerly initializing database pools and SDK configurations during cold boot. --- # 148. LAZY INIT Deferring initialization shifts overhead onto the first incoming user request. Document the tradeoff. --- # 149. CONTAINER STARTUP Container startup and health check readiness speed determines auto-scaling responsiveness during sudden traffic spikes. --- # 150. AUTOSCALING Review scaling trigger metrics: - CPU utilization - request concurrency - custom queue depth If scaling triggers lag behind traffic bursts, latency spikes occur. --- # 151. AUTOSCALING IS NOT MAGIC Database instances and external APIs do not automatically scale in lockstep with application replicas. --- # 152. SHARED BOTTLENECK Adding application instances provides zero relief if database CPU utilization is already pinned at 100%. --- # 153. HORIZONTAL SCALING Identify shared state preventing horizontal scaling: - local in-memory sessions - local filesystem writes - process-level locks - in-process memory queues --- # 154. STICKY SESSION Session stickiness can be legitimate, but skews load balancing and hampers failure failover. --- # 155. VERTICAL SCALING Upgrading instance compute or memory is often the most pragmatic and cost-effective solution for specific workloads. Never dismiss vertical scaling simply because it is not "cloud-native." --- # 156. RATE LIMIT / LOAD SHEDDING When systems reach peak capacity: gracefully shedding excess load via 429s or 503s is far better than suffering total systemic collapse. --- # 157. QUEUE ADMISSION Never accept unbounded incoming work when downstream processing capacity is exhausted. --- # 158. BACKPRESSURE Producers must receive backpressure signals when consumers lag behind. --- # 159. BOUNDED QUEUE Bounded queues prevent memory exhaustion, but reject semantics must be well-defined. --- # 160. LOAD TEST When reviewing load tests inspect: - workload distribution models - concurrency levels - test duration - data volume realism - user think times - ramp-up patterns --- # 161. 1-SECOND BENCHMARK A 1-second benchmark cannot surface: - memory leaks - connection pool depletion - cache warming behaviors - steady-state thermal and GC dynamics --- # 162. SOAK TEST Extended soak testing uncovers: - slow memory leaks - connection leaks - queue backlog drift --- # 163. SPIKE TEST Sudden step-function traffic increases. --- # 164. STRESS TEST Push systems until reaching physical breaking points. --- # 165. BREAKPOINT Establish: > At what exact concurrency or throughput does the system begin to degrade? --- # 166. PERFORMANCE CLIFF Systems often perform predictably until hitting a sharp performance cliff caused by: - pool starvation - database lock contention - GC thrashing - downstream saturation --- # 167. REALISTIC DATA A test database with 100 rows is unrepresentative if production houses 10 million records. --- # 168. DATA DISTRIBUTION Query execution plans change dramatically based on data skew and column cardinality. --- # 169. HOT TENANT A massive tenant can display performance characteristics radically different from average tenants. --- # 170. MULTI-TENANT FAIRNESS A single noisy tenant must not degrade service for all others where isolation is expected. --- # 171. NOISY NEIGHBOR Applies across: - background queues - database resources - external API quotas --- # 172. PER-TENANT LIMITS Per-tenant rate limiting and capacity caps protect overall system health, but require product alignment. --- # 173. LOAD GENERATOR BOTTLENECK Verify that the load testing tool itself did not saturate its own CPU or network interface during testing. --- # 174. NETWORK LOCATION Testing from the same local datacenter does not measure real-world internet routing latency. Differentiate: ```text server processing latency ``` from: ```text end-to-end user latency ``` --- # 175. TTFB Where user web performance depends on Time to First Byte: break down backend processing versus network transit time. --- # 176. P95/P99 Tail latency governs interactive user experience. --- # 177. OUTLIERS Do not discard latency outliers as mere "noise." They often highlight: - cold starts - lock waits - GC pauses - cache misses --- # 178. COORDINATED OMISSION Verify whether load testing harnesses hide queuing latency during system overload. Do not theorize if testing tools are unverified. --- # 179. WARMUP JIT compilation and cache pre-warming distort early test measurements. --- # 180. DEBUG MODE Benchmarking must always utilize production release builds and configurations. --- # 181. PROFILER OVERHEAD Active profilers introduce measurement overhead. --- # 182. PRODUCTION SAMPLE Low-overhead continuous profiling in production provides the highest-fidelity evidence. --- # 183. COST Performance and infrastructure costs are inextricably linked: - database IOPS - network egress - compute instances - cache memory - message queue operations --- # 184. OVERPROVISIONING A faster system achieved by 10x over-provisioning compute is rarely optimal. --- # 185. COST PER REQUEST Where metrics exist: tracks efficiency across architectural changes. --- # 186. CACHE COST Managed Redis clusters and CDN bandwidth carry financial costs. Ensure caching yields a measurable return on investment. --- # 187. DATABASE READ REPLICA Read replicas expand read capacity. However, replica lag can compromise business correctness. --- # 188. SHARDING Never recommend database sharding without demonstrating that a single database engine cannot meet scale requirements. --- # 189. PARTITIONING Valuable for massive time-series or historical audit tables. --- # 190. ARCHIVAL Archiving cold historical data out of hot transactional tables improves operational efficiency. Product query requirements dictate viability. --- # 191. MATERIALIZED VIEW Accelerates heavy aggregations, but introduces data refresh and maintenance tradeoffs. --- # 192. PRECOMPUTATION Ideal when multiple consumers require identical expensive calculation results. --- # 193. ASYNC GENERATION Reports and exports can be precomputed or queued asynchronously. --- # 194. PERFORMANCE VS FRESHNESS Serving cached or precomputed data yields higher speed at the cost of data freshness. Document this tradeoff explicitly. --- # 195. PERFORMANCE VS CONSISTENCY Read replicas, caching, and eventual consistency must never compromise critical business logic. --- # 196. PERFORMANCE VS SECURITY Never disable: - authentication checks - payload encryption - password hashing cost factors - input validation for the sake of speed. --- # 197. PERFORMANCE VS RELIABILITY Do not ramp up concurrency to levels that invite retry storms and cascading failures. --- # 198. PERFORMANCE VS COMPLEXITY A 100 ms speedup rarely justifies the complexity of a distributed cache or event broker on low-traffic endpoints. --- # 199. FINDING FORMAT Every serious finding must include: ```text ID: Severity: Category: Confidence: Status: Endpoint/Job: Service: File/Class: Function: Resource: Workload: Concurrency: Dataset: Environment: Problem: Evidence: Hot Path: T0: T1: T2: T3: Measured latency: P50: P95: P99: If unavailable: NOT MEASURED Resource utilization: Bottleneck: Scaling behavior: User impact: Capacity impact: Cost impact: Root cause: Recommended remediation: Expected benefit: HIGH / MEDIUM / LOW / NOT MEASURED Benchmark / load test: Verification after fix: Complexity: XS / S / M / L / XL ``` --- # 200. SEVERITY Use: ## P0 - CRITICAL Reserved strictly for performance flaws directly triggering: - systemic outages with critical impact - catastrophic resource exhaustion - critical data loss or security failures ## P1 - HIGH - critical endpoints routinely breach acceptable latency or capacity targets - load triggers total systemic collapse - connection or resource exhaustion crashes services - queue backlogs grow unbounded under normal production load ## P2 - MEDIUM - significant latency or throughput bottleneck - important endpoints perform verified redundant expensive work ## P3 - LOW - minor localized hotspot with limited system impact ## P4 - IMPROVEMENT - optimization opportunities without confirmed user or capacity issues --- # 201. CONFIDENCE Use: ```text HIGH MEDIUM LOW ``` HIGH: production traces, query plans, or load tests directly prove the bottleneck. MEDIUM: strong static code or configuration evidence, but workload scale is not fully verified. LOW: depends on unverified traffic volumes or unknown dataset dimensions. --- # 202. STATUS Use: ```text CONFIRMED LIKELY THEORETICAL NOT VERIFIED ``` --- # 203. PERFORMANCE EVIDENCE Indicate: ```text PRODUCTION METRIC PRODUCTION TRACE LOAD TEST PROFILER QUERY PLAN BENCHMARK STATIC CODE NONE ``` --- # 204. FALSE-POSITIVE PREVENTION Before any P1/P2 finding verify: 1. invocation frequency 2. production or load-test evidence 3. dataset volume 4. concurrency levels 5. caching behavior 6. database query plans 7. downstream latency figures 8. release build configuration 9. infrastructure topology 10. current resource utilization --- # 205. DO NOT OPTIMIZE O(N) JUST BECAUSE IT IS O(N) If N <= 20: the algorithmic complexity is almost certainly irrelevant. --- # 206. DO NOT ADD CACHE AUTOMATICALLY Caching introduces: - cache invalidation complexity - stale state bugs - memory footprint - additional infrastructure dependencies --- # 207. DO NOT ADD INDEX TO EVERY COLUMN Index additions must be strictly driven by query patterns. --- # 208. DO NOT PARALLELIZE EVERYTHING Parallelizing work often overwhelms databases and upstream dependencies. --- # 209. DO NOT INCREASE POOLS BLINDLY Enlarging thread or connection pools simply shifts the bottleneck downstream. --- # 210. DO NOT REWRITE IN ANOTHER LANGUAGE Switching languages (e.g. Node to Go/Rust) is not a performance fix without demonstrating an unavoidable runtime bottleneck. --- # 211. DO NOT INTRODUCE MICROSERVICES FOR PERFORMANCE Network boundaries almost always inflate latency. --- # 212. DO NOT INTRODUCE REDIS FOR PERFORMANCE First prove the existence of repeated, expensive read operations. --- # 213. DO NOT SACRIFICE CORRECTNESS Serving 5-minute stale account balances is unacceptable if business domains require real-time accuracy. --- # 214. DO NOT MODIFY CODE During audit: - do not add caches - do not add indexes - do not alter pool sizing - do not change concurrency settings - do not introduce message queues - do not rewrite queries Complete the audit first. --- # 215. OUTPUT - BACKEND_PERFORMANCE_AUDIT.md Structure the final report: ## 1. Executive Summary - stack - topology - performance evidence - critical bottlenecks - capacity risks ## 2. Performance Measurement Coverage ## 3. Critical Endpoint Map ## 4. Latency Breakdown ## 5. CPU Audit ## 6. Event Loop / Thread Audit ## 7. Database Performance ## 8. Query / Index Audit ## 9. Connection Pool Audit ## 10. Transaction / Lock Contention ## 11. Cache Audit ## 12. Serialization / Payload Audit ## 13. File / Streaming Audit ## 14. External Dependency Latency ## 15. Retry / Timeout Load Amplification ## 16. Queue / Worker Performance ## 17. Memory / GC Audit ## 18. Logging / Observability Overhead ## 19. Serverless / Startup Performance ## 20. Horizontal Scaling Audit ## 21. Backpressure / Load Shedding ## 22. Load Test Coverage ## 23. Cost / Efficiency ## 24. Findings Summary | ID | Severity | Component | Bottleneck | Evidence | Confidence | Status | |---|---|---|---|---|---|---| ## 25. P1 Findings ## 26. P2 Findings ## 27. P3 Findings ## 28. P4 Improvements ## 29. Things Done Well ## 30. Unknown / Not Measured ## 31. Optimization Roadmap --- # 216. ENDPOINT PERFORMANCE MATRIX | Endpoint | RPS | P50 | P95 | P99 | DB calls | External calls | |---|---:|---:|---:|---:|---:|---:| If data is unavailable: **NOT MEASURED** --- # 217. DATABASE MATRIX | Query | Calls/request | Rows | Plan | Duration | Risk | |---|---:|---:|---|---:|---| --- # 218. RESOURCE MATRIX | Resource | Limit | Typical | Peak | Saturation risk | |---|---:|---:|---:|---| For: - CPU - memory - DB pool - worker pool - queue - sockets --- # 219. DEPENDENCY MATRIX | Dependency | Calls | P95 | Timeout | Retry | Critical | |---|---:|---:|---:|---|---| --- # 220. CACHE MATRIX | Cache | Hit rate | TTL | Miss cost | Size | Risk | |---|---:|---:|---:|---:|---| --- # 221. QUEUE MATRIX | Queue | Produce rate | Consume rate | Depth | Oldest age | Risk | |---|---:|---:|---:|---:|---| --- # 222. SECOND PASS - TOP LATENCY ATTACK Select the slowest critical endpoints based on available telemetry. Deconstruct each into its slowest dependency, query, or CPU phase. --- # 223. SECOND PASS - P99 ATTACK Do not focus solely on median numbers. Ask: > What differentiates the slowest 1% of requests? Potential causes: - cache misses - lock waits - cold starts - GC pauses - downstream timeouts - bad query execution plans --- # 224. SECOND PASS - 10X TRAFFIC Mentally or via load testing evaluate: ```text current traffic × 10 ``` Determine which resource saturates first. --- # 225. SECOND PASS - 10X DATA For critical queries evaluate: ```text current rows × 10 ``` Inspect query plan degradation. --- # 226. SECOND PASS - CACHE MISS STORM Assume the cache suddenly empties. Can the origin database survive the sudden load? --- # 227. SECOND PASS - DOWNSTREAM SLOW External APIs slow down by 10x. Examine: - active request counts - connection pool saturation - retry amplification - memory consumption --- # 228. SECOND PASS - DB SLOW Database query latency surges: ```text 20 ms -> 2 s ``` Assess the impact on: - connection pools - server concurrency - retries - job queues --- # 229. SECOND PASS - POOL SATURATION Assume all database connections are in use. Ask: > How long do incoming requests wait, and how do they fail? --- # 230. SECOND PASS - WORKER BACKLOG Arrival rates exceed consumption rates over a 1-hour window. Calculate backlog accumulation. --- # 231. SECOND PASS - PROCESS MEMORY Sustain workload over extended periods. Determine: - does heap stabilize - does RSS climb - do cache entries grow unbounded - do buffer allocations leak --- # 232. SECOND PASS - LARGE REQUEST Dispatch maximum permissible payloads and files. Map where redundant memory copies occur. --- # 233. SECOND PASS - LARGE RESPONSE Evaluate the largest realistic export or list queries: - database memory - application heap - serialization overhead - compression cycles - network transit time --- # 234. SECOND PASS - RETRY STORM Simulate dependency outages with active retry policies. Calculate total downstream requests generated per incoming call. --- # 235. SECOND PASS - COLD START In serverless or autoscaling container environments: measure initial cold start latency separately from warm invocations. --- # 236. SECOND PASS - LOW CACHE HIT Drop cache hit rate assumptions. Verify whether database headroom accommodates the increased traffic. --- # 237. SECOND PASS - HOT TENANT One tenant contains 100x more data than the average. Verify whether queries and endpoints remain performant. --- # 238. SECOND PASS - CRON SPIKE Execute scheduled batch jobs concurrently with peak user traffic. Look for shared database and CPU resource contention. --- # 239. SECOND PASS - FAILURE UNDER LOAD Terminate downstream services under heavy load. Determine whether the system: - degrades gracefully - collapses in cascading failure --- # 240. FINAL QUALITY GATE Before final response verify: - optimization recommendations do not precede hotspot evidence - production topology is considered - P50, P95, and P99 latency numbers are distinguished where available - averages are not used as the sole latency metric - critical endpoints have mapped hot paths - total database query counts are analyzed, not just single-query speed - N+1 issues are verified via actual execution flows - index suggestions are backed by query plans or sound query logic - connection pool sizing accounts for total instance counts - long transactions and lock contention are investigated - cache suggestions include expected hit rates and miss costs - correctness is not sacrificed for raw speed - external calls include sequential, parallel, and fan-out analysis - retries are evaluated as load multipliers - queue produce and consume rates are analyzed - memory leaks are separated from legitimate high memory utilization - debug and profiling measurements are not conflated with production performance - load tests use realistic data volumes and workloads where feasible - horizontal scaling is not assumed to resolve shared database bottlenecks - performance fixes never weaken security or business invariants - micro-optimizations are classified as P4 or omitted --- # FINAL RULE I do not want a report like: > Add Redis, indexes, pagination, and horizontal scaling. That is not a backend performance audit. I am looking for problems like: ```text GET /dashboard ↓ loads 100 projects ↓ for each project executes 3 additional queries ↓ 301 SQL queries per request ↓ P95 grows sharply with user project count ``` or: ```text backend has 20 replicas ↓ each has DB pool max 30 ↓ possible DB connections = 600 ↓ database max connections = 200 ↓ traffic spike ↓ connection storm / timeouts ``` or: ```text external API normally 200 ms ↓ outage makes calls last 10 s ↓ backend timeout = 30 s ↓ each incoming request holds resources ↓ automatic retry doubles calls ↓ capacity collapses ``` or: ```text cache key expires ↓ 500 concurrent requests miss ↓ all independently execute same expensive DB aggregation ↓ database CPU spikes ↓ latency rises across unrelated endpoints ``` or: ```text export endpoint ↓ loads 2 million rows into memory ↓ maps to DTO list ↓ serializes full JSON/CSV in memory ↓ multiple copies coexist ↓ process OOM ``` or: ```text Node.js request handler ↓ synchronous image compression ↓ CPU busy for 700 ms ↓ event loop blocked ↓ all other requests on instance wait ``` or: ```text queue producer = 100 jobs/sec worker capacity = 80 jobs/sec ↓ backlog grows by 20/sec ↓ 72,000 additional jobs after one hour ↓ oldest job latency continuously increases ``` These are the backend performance problems you need to find. Think through: - latency budgets - invocation frequency - request concurrency - dataset dimensions - physical resource limits - database query plans - downstream latency profiles - queue throughput dynamics - cache hit and miss tradeoffs - tail latency distribution - resource saturation For each serious finding you must be able to answer: > Where is the actual bottleneck located? > How frequently is the execution path exercised? > What is the dataset size? > Which physical resource saturates? > What happens to P95/P99 latency under increased load? > Does the proposed fix accelerate the bottleneck itself or merely surrounding code? > How will performance be measured before and after the change? If unmeasured: **NOT MEASURED.** If strong static code signals exist without benchmarks: **CODE-LEVEL BOTTLENECK.** If merely a possible optimization: **P4 - IMPROVEMENT.** It is better to find 5 genuine hot path bottlenecks than to write 100 micro-optimizations. The goal is to produce a forensically precise backend performance audit that can be directly converted into: - profiler investigations - query-plan analyses - load test scenarios - reproducible benchmarks - capacity planning models - targeted optimizations - pre/post validation measurements - production performance verification
分类与标签
编程开发、后端性能、数据库查询、负载测试、容量规划