后台任务与队列可靠性审计
追踪任务从入队到完成的全生命周期,形成有证据的丢失、重复执行与恢复缺陷报告。
完整提示词
# BACKGROUND JOBS AND QUEUE AUDIT I want you to perform a maximally deep, systematic, evidence-first, and production-oriented analysis of the entire background job and queue system of the backend application. Main objective: > Determine whether background jobs reliably survive process restarts, retries, duplicate deliveries, worker crashes, queue outages, deployments, concurrency, partial failures, and backlogs without job loss, duplicate side effects, endless retries, stuck jobs, or corrupted business state. This is not: - generic advice to use Redis/BullMQ/SQS/RabbitMQ/Kafka - automatically offloading every slow HTTP request into a queue - a superficial check to verify whether a worker process is running - an assumption that every job runs exactly once - an assumption that retrying blindly solves reliability problems - automatically increasing worker concurrency limits - solely a performance audit - solely an error-handling audit The focus is on the complete job lifecycle: ```text job creation ↓ durable enqueue ↓ queue storage ↓ worker claim ↓ processing ↓ side effects ↓ ack / completion ↓ retry / terminal state ``` Priority: **job durability > idempotency > business correctness > retry safety > concurrency > recovery > backlog control > performance** It is better to discover 6 real job-loss, duplicate execution, or stuck-worker problems than to produce 100 generic queue recommendations. --- # 1. DETERMINE JOB STACK Before filing findings, establish: - queue technology - broker and storage engine - worker framework - scheduler implementation - cron engine - delayed job mechanics - retry mechanism - dead-letter queue (DLQ) mechanism - persistence configuration - concurrency model - deployment topology - observability tooling Examples: ```text BullMQ Redis Celery RabbitMQ SQS Sidekiq Hangfire Quartz WorkManager-like backend workers custom DB queue ``` Do not assume the technology stack in advance. --- # 2. INVENTORY ALL JOBS Identify: - background jobs - scheduled jobs - cron tasks - delayed jobs - queue consumers - event consumers - batch processing tasks - maintenance tasks - cleanup routines - export generators - transactional emails - push notifications - billing runs - reconciliation tasks For each, document: ```text Job name: Trigger: Queue: Payload: Durable: Retry: Max attempts: Timeout: Concurrency: Side effects: Business criticality: ``` --- # 3. JOB CLASSIFICATION Classify each job: ```text CRITICAL IMPORTANT BEST-EFFORT MAINTENANCE ANALYTICS ``` Severity must reflect real-world business and financial impact. --- # 4. JOB CREATION FLOW For each job, map: ```text request/event ↓ business commit ↓ enqueue ``` Ask: > What happens if the enqueue call fails? --- # 5. DUAL WRITE Classic vulnerability: ```text DB write succeeds ↓ enqueue fails ``` The database state asserts work should happen, but the background job record is lost. --- # 6. ENQUEUE BEFORE DB COMMIT Inverse flaw: ```text job enqueued ↓ worker starts ↓ DB transaction not committed yet ``` The worker queries the database and finds a missing or partial state. --- # 7. OUTBOX If a job is triggered by a database state change and must be guaranteed to run: verify the presence of: - a transactional outbox table - a durable work queue table - an equivalent atomic mechanism Do not mandate an outbox for low-value, best-effort notifications. --- # 8. DURABLE ENQUEUE Pinpoint the precise milestone where a job becomes truly durable on persistent storage. --- # 9. IN-MEMORY JOB If the system uses: ```text setTimeout in-memory array thread executor only ``` for work that must survive a process restart: this is a genuine reliability failure. --- # 10. SERVERLESS BACKGROUND WORK Pattern: ```text return HTTP response ↓ continue async in same process ``` is fundamentally unreliable on serverless platforms. Verify the specific hosting runtime constraints. --- # 11. JOB IDENTITY Every critical job must have a deterministic way to be uniquely identified. --- # 12. JOB ID VS BUSINESS OPERATION ID Distinguish: ```text job attempt ID ``` from: ```text logical business operation ID ``` A single logical operation can generate multiple execution attempts. --- # 13. PAYLOAD Map what the job payload encapsulates: - full domain object - resource entity ID - point-in-time snapshot - mutable object references --- # 14. STALE PAYLOAD If the job carries a historical snapshot of mutable state: it may apply stale data after the underlying entity has already changed. --- # 15. RESOURCE ID PAYLOAD If the job fetches current state at execution time: this prevents stale state for mutable entities. However, it is unsuitable if an immutable historical audit snapshot was required. --- # 16. SNAPSHOT VS CURRENT STATE Explicitly determine whether each job requires an immutable snapshot or current state. --- # 17. PAYLOAD SIZE Bloated payloads degrade: - queue broker memory - serialization overhead - network transit time - worker heap consumption --- # 18. SENSITIVE PAYLOAD Do not store: - raw passwords - API access tokens - full payment credentials in queue payloads without justification. --- # 19. SERIALIZATION Verify: - schema format - schema versioning - type preservation - backward compatibility --- # 20. DEPLOYMENT VERSION SKEW Scenario: ```text old producer ↓ new worker ``` or: ```text new producer ↓ old worker ``` during rolling zero-downtime deployments. --- # 21. JOB PAYLOAD VERSIONING If jobs can linger in backlogs across deployments: workers must gracefully parse payloads emitted by older application builds. --- # 22. CLASS NAME SERIALIZATION If the queue framework serializes fully qualified class or function names: renaming or moving classes during deployment will crash pending jobs. --- # 23. UNKNOWN JOB TYPE Unknown job types should not be silently dropped without alerting. --- # 24. WORKER CLAIM Determine the locking or lease acquisition mechanism used by workers. --- # 25. VISIBILITY TIMEOUT If the queue relies on visibility timeouts or leases: the timeout must be substantially longer than, or actively renewed during, real job execution. --- # 26. JOB EXCEEDS VISIBILITY TIMEOUT Scenario: ```text worker A processes job ↓ visibility expires ↓ worker B receives same job ↓ both run simultaneously ``` Critical duplicate execution risk. --- # 27. LEASE RENEWAL If heartbeat or lease renewal is used: verify failure detection and restart semantics. --- # 28. ACK Determine when the queue acknowledges and marks a job complete. --- # 29. ACK BEFORE SIDE EFFECT Critical defect: ```text ack ↓ process crashes ↓ side effect never happens ``` The job is permanently lost. --- # 30. ACK AFTER SIDE EFFECT Standard pattern, but carries duplicate delivery risks: ```text side effect succeeds ↓ process crashes before ack ↓ job redelivered ``` Duplicate execution occurs on restart. --- # 31. AT-LEAST-ONCE Most message queues enforce at-least-once delivery, requiring workers to tolerate duplicate runs. Do not claim exactly-once processing without end-to-end evidence. --- # 32. IDEMPOTENCY For each critical job, ask: > What happens if the exact same logical job executes twice? --- # 33. IDEMPOTENT DB UPDATE Example: ```text set status = PROCESSED ``` is naturally idempotent. --- # 34. NON-IDEMPOTENT SIDE EFFECT Example: ```text send money increment credit send email create invoice ``` requires defensive guards calibrated to financial and business risk. --- # 35. DEDUP If job deduplication exists: verify: - deduplication key - key TTL - persistence mechanism - cross-instance enforcement --- # 36. DEDUP IS NOT IDEMPOTENCY Deduplication prevents enqueuing duplicate items. It does not protect against: ```text side effect succeeded ↓ worker crashed before ack ↓ same job redelivered ``` --- # 37. UNIQUE BUSINESS OPERATION Critical business mutations should enforce unique operation keys in the database. --- # 38. JOB RETRY Inventory: - max retry attempts - backoff algorithm - jitter configuration - retryable error definitions --- # 39. RETRY ALL Anti-pattern: ```text catch every exception ↓ retry ``` Certain errors are permanent and will never resolve via retry. --- # 40. VALIDATION ERROR An invalid payload schema will not become valid on retry. --- # 41. NOT FOUND Can represent: - a transient replication race - a permanent entity deletion - an expected business branch Analyze based on domain semantics. --- # 42. AUTH ERROR Third-party authentication failures typically require credential rotation or config updates, not fast retry loops. --- # 43. 429 Apply exponential backoff honoring upstream rate-limit headers. --- # 44. TIMEOUT May be transient, but downstream mutation results remain unknown. --- # 45. 5XX Bounded retries with backoff are generally appropriate. --- # 46. RETRY CLASSIFICATION Categorize errors: ```text TRANSIENT PERMANENT BUSINESS REJECTION UNKNOWN OUTCOME CANCELLED ``` --- # 47. NESTED RETRIES Map retry multiplication: ```text queue retries ↓ service retries ↓ HTTP client retries ``` Total attempts can easily multiply out of control. --- # 48. RETRY STORM An external provider outage combined with hundreds of jobs causes synchronized retry storms upon recovery. --- # 49. JITTER Essential for large worker fleets and deep queues. --- # 50. MAX ATTEMPTS Never retry indefinitely without explicit business justification. --- # 51. MAX AGE Certain jobs expire after a defined time window. Example: - push notifications - time-sensitive reminders --- # 52. DEAD LETTER Permanent failures must transition to a terminal state or dead-letter queue (DLQ) where required. --- # 53. FAILED JOB VISIBILITY Failed jobs must not vanish silently into unmonitored tables. --- # 54. POISON JOB A permanently crashing job must not stall an entire queue. --- # 55. FIFO QUEUE If a poison job blocks an entire ordered partition or FIFO group: verify error isolation and unblocking procedures. --- # 56. MANUAL RETRY If an administrator triggers a manual retry: verify: - same logical operation ID - safety of re-executing side effects - comprehensive audit logging --- # 57. RETRY AFTER PARTIAL SUCCESS The most critical distributed failure scenario. --- # 58. STEP 1 SUCCESS, STEP 2 FAIL Example: ```text upload file ↓ DB insert fails ↓ job retries ↓ file uploaded again ``` --- # 59. CHECKPOINTING Multi-step batch jobs often require durable step checkpointing. Do not impose checkpointing if the entire job can simply be designed idempotent from the start. --- # 60. PARTIAL STATE A retried job must detect which execution stages were already completed. --- # 61. COMPENSATION If a job must roll back earlier side effects after a downstream failure: verify compensation logic. --- # 62. COMPENSATION FAILURE What occurs if the compensating rollback action also fails? --- # 63. CONCURRENCY Determine worker concurrency boundaries: - per process - cluster-wide - per queue - per tenant - per resource --- # 64. CONCURRENCY > 1 Ask whether two parallel jobs can mutate the same entity concurrently. --- # 65. RESOURCE CONFLICT Scenario: ```text Job A updates order Job B updates same order ``` --- # 66. SERIALIZATION PER RESOURCE If ordering must be strictly preserved per entity: verify queue partitioning, consumer groups, or lock keys. --- # 67. GLOBAL SERIALIZATION Running a single global worker preserves ordering but destroys throughput. Do not recommend global serialization without strict necessity. --- # 68. PARALLEL SAFE Independent, embarrassingly parallel tasks should run concurrently. --- # 69. UNBOUNDED WORKER CONCURRENCY Can saturate: - database connection pools - third-party API quotas - host CPU - available memory --- # 70. DOWNSTREAM CAPACITY Worker concurrency must align with the capacity of the bottleneck dependency. --- # 71. CONCURRENCY LIMIT Do not increase concurrency blindly just because backlogs grow. First identify why consumption is bottlenecked. --- # 72. QUEUE BACKLOG Track: - queue depth - oldest job age - processing latency --- # 73. DEPTH IS NOT ENOUGH 100,000 one-millisecond jobs differ vastly from 100 ten-minute jobs. --- # 74. OLDEST JOB AGE Often the most reliable indicator of user-perceived processing delay. --- # 75. PRODUCER RATE Measure: ```text jobs/sec produced ``` --- # 76. CONSUMER RATE Measure: ```text jobs/sec completed ``` --- # 77. NEGATIVE CAPACITY If: ```text produce rate > consume rate ``` sustained over time: backlog expands uncontrollably. --- # 78. BURST Queues are designed to absorb short-lived ingestion spikes. Temporary backlogs are expected behavior if they drain normally. --- # 79. SUSTAINED OVERLOAD If the backlog never decreases during normal operations: this is a fundamental capacity failure. --- # 80. AUTOSCALING WORKERS If autoscaling is deployed: verify the driving metric signal. --- # 81. SCALE ON DEPTH Scaling solely on raw depth can over-provision for large, slow jobs. --- # 82. SCALE ON OLDEST AGE Often provides a more stable scaling trigger for latency-sensitive workloads. --- # 83. SCALE LIMIT Downstream dependency capacity must define hard ceilings for worker autoscaling. --- # 84. THUNDERING HERD Scaling 100 new worker pods simultaneously can immediately take down the database or downstream APIs. --- # 85. QUEUE PRIORITY If multiple priority levels are used: verify anti-starvation mechanisms. --- # 86. HIGH PRIORITY STARVATION An endless stream of high-priority jobs can permanently starve low-priority tasks. --- # 87. FAIRNESS A single abusive tenant or user can saturate an entire shared queue. --- # 88. PER-TENANT BACKLOG In multi-tenant systems: audit noisy-neighbor isolation controls. --- # 89. QUEUE PARTITION Partitioning queues aids tenant isolation, but avoid over-engineering without product requirements. --- # 90. SCHEDULED JOBS Inventory all cron and scheduled recurring tasks. --- # 91. DUPLICATE CRON If every backend app replica executes: ```text cron every minute ``` the identical scheduled job triggers concurrently across all replicas. --- # 92. PLATFORM SCHEDULER If the cloud platform natively guarantees single-instance execution: do not report duplicate cron runs without verification. --- # 93. CRON IDEMPOTENCY Even singleton schedulers can re-fire or retry after transient network disconnects. --- # 94. LONGER THAN INTERVAL Scenario: ```text cron every 5 min job duration = 10 min ``` Do executions overlap concurrently? --- # 95. REENTRANCY If a scheduled job must not execute concurrently with itself: enforce distributed mutex or overlap prevention locks. --- # 96. MISFIRE What happens if the scheduler is down for 2 hours? - execute once - execute all missed runs - skip entirely --- # 97. BACKFILL STORM If a recovered scheduler fires all missed intervals at once: it can cause severe infrastructure overload. --- # 98. TIMEZONE Scheduling cron jobs in local time zones introduces daylight saving time (DST) edge cases. --- # 99. DST DUPLICATE The repeated hour during fall DST transitions can trigger scheduled runs twice. --- # 100. DST SKIP The lost hour during spring DST transitions can skip scheduled executions completely. --- # 101. MONTH-END Jobs anchored to calendar boundaries require rigorous validation. --- # 102. DELAYED JOB Verify: - scheduling precision - persistence guarantees - restart behavior - time source drift --- # 103. CANCEL DELAYED JOB If the target entity is deleted or cancelled before execution: the job must verify domain state before proceeding. --- # 104. STALE JOB Scenario: ```text job scheduled while ACTIVE ↓ resource becomes CANCELLED ↓ old job executes ↓ side effect still runs ``` --- # 105. CURRENT STATE CHECK Delayed jobs must never blindly trust stale payload data when business rules require current entity state. --- # 106. REMINDERS A scheduled reminder may become invalid after an intermediate state transition. --- # 107. EXPIRATION JOBS If entity expiration relies solely on delayed jobs: what happens if queue delays postpone execution by 30 minutes? --- # 108. STATE DERIVATION Expiration should ideally be evaluated dynamically based on timestamps rather than relying exclusively on physical queue execution. Do not refactor without domain understanding. --- # 109. BATCH JOBS Inventory jobs designed to process large volumes of records. --- # 110. LOAD ALL Pattern: ```text SELECT 1,000,000 rows ↓ process all ``` triggers memory exhaustion and execution timeouts. --- # 111. CHUNKING Batch processing must stream or page records in manageable chunks. --- # 112. CHECKPOINT If a job crashes after processing record 900,000: does the retry re-execute from record 0? --- # 113. DUPLICATE PREVIOUS ITEMS If processing side effects are non-idempotent: restarting batch jobs from the beginning is destructive. --- # 114. PER-ITEM STATE May be necessary for long-running, critical batch jobs. --- # 115. TRANSACTION SIZE Do not wrap one million record updates inside a single database transaction without justification. --- # 116. PARTIAL BATCH SEMANTICS Explicitly define: - all-or-nothing - partial progress with progress markers - individual item retry --- # 117. FAILED ITEM A single malformed item should not block 999,999 valid records if domain rules permit partial success. --- # 118. ERROR ISOLATION Isolate poison records into a failure collection. --- # 119. BATCH CONCURRENCY Running excessive batch chunks in parallel saturates database I/O. --- # 120. EXTERNAL API JOB For workers integrating with third-party APIs: verify: - HTTP timeouts - retry policies - rate limit compliance - request idempotency keys --- # 121. PROVIDER QUOTA Worker concurrency must not exceed external API quotas without throttling. --- # 122. 429 Queues should throttle or delay retries rather than magnifying retry storms upon receiving 429 responses. --- # 123. EMAIL JOB Transactional email dispatch is a classic queue workload. Audit duplicate email transmission risks. --- # 124. SMS JOB Same as email, but carries immediate financial billing costs. --- # 125. PAYMENT JOB High risk. Scrutinize unknown network timeout outcomes and payment gateway idempotency keys. --- # 126. EXPORT JOB Map lifecycle: ```text queued ↓ processing ↓ file generation ↓ storage ↓ ready ``` --- # 127. EXPORT DUPLICATION Retries can generate redundant export files and orphan artifacts. --- # 128. ORPHAN FILE File is uploaded to object storage, but the database update fails. --- # 129. STALE EXPORT Data can mutate during multi-minute export generations. Determine whether the business requires: - an immutable point-in-time snapshot - eventual current state --- # 130. IMPORT JOB Scrutinize partial failure modes and idempotency for file imports. --- # 131. MEDIA PROCESSING Video, image, and transcoding jobs carry: - CPU/GPU bounds - temporary disk space consumption - extended execution durations --- # 132. TEMP CLEANUP Process crashes must not leave unbounded temporary files on worker disks. --- # 133. RESOURCE LIMIT Workers must not spawn more transcoding processes than physical hardware can handle. --- # 134. RECONCILIATION JOB Reconciliation tasks repairing data inconsistencies must be exceptionally conservative. --- # 135. RECONCILIATION IDEMPOTENCY Repeated execution of reconciliation routines must never corrupt valid data. --- # 136. CLEANUP JOB Purging historical records, cached files, or data partitions must enforce strict retention boundaries. --- # 137. CLEANUP RACE An entity might transition to active right as the cleanup routine marks it eligible for deletion. --- # 138. DELETE BY AGE Verify: - UTC timezone consistency - precise cutoff boundaries - active foreign key and domain references --- # 139. RETENTION Never hardcode retention windows without consulting legal and product compliance policies. --- # 140. JOB CANCELLATION If users can abort active tasks: map cancellation mechanics. --- # 141. CANCEL PENDING Straightforward state transition before worker acquisition. --- # 142. CANCEL RUNNING Complex. Workers must implement cooperative cancellation tokens if supported by the runtime. --- # 143. SIDE EFFECT BEFORE CANCEL Cancellation cannot undo external side effects that were already executed. --- # 144. TERMINAL STATE Common terminal job states: ```text COMPLETED FAILED CANCELLED ``` --- # 145. CANCEL RACE Scenario: ```text user cancels ↓ worker completes at same moment ``` Determine which terminal state wins deterministically. --- # 146. STATUS UPDATE ATOMICITY Terminal state transitions must be atomic and race-free. --- # 147. JOB TIMEOUT Establish explicit per-job execution timeouts. --- # 148. TIMEOUT IS NOT CANCEL Framework timeouts may mark the job failed while the underlying thread continues running. Verify runtime semantics. --- # 149. HARD KILL If worker processes are abruptly terminated (e.g., OOM killer): the queue must safely detect and redeliver orphaned jobs. --- # 150. GRACEFUL SHUTDOWN Upon receiving SIGTERM/SIGINT during deployments: - stop claiming new jobs - allow active jobs to finish or cancel cooperatively - release leases or un-ack appropriately according to framework specifications. --- # 151. DEPLOYMENT Workers restart frequently during rolling application releases. This is standard production operation, not an edge case. --- # 152. JOB DURING DEPLOYMENT Test behavior when long-running jobs are interrupted by a worker shutdown. --- # 153. CODE VERSION CHANGE Pending jobs were serialized by older application code. New workers must deserialize old payloads or provide data migration paths. --- # 154. REMOVED WORKER Never remove a worker class while unconsumed jobs of that type remain in queues. --- # 155. RENAMED QUEUE Deploying a queue name change can abandon pending jobs in old queues without consumers. --- # 156. ORPHAN QUEUE Search for queue names defined in brokers that have zero active consumers. --- # 157. PRODUCER WITHOUT CONSUMER Critical operational failure finding. --- # 158. CONSUMER WITHOUT PRODUCER May represent dead legacy code or abandoned features. --- # 159. CONFIG DRIFT Producers and workers must not disagree on: - queue names - broker connection URLs - Redis prefixes/namespaces --- # 160. ENVIRONMENT LEAK Production producers must never push jobs into staging queues, and vice versa. --- # 161. PREFIX/NAMESPACE If dev, staging, and production share a broker: queue namespaces must be strictly isolated. --- # 162. CROSS-ENVIRONMENT JOB Classify as P1/P0 depending on data leakage and mutation impact. --- # 163. MULTI-TENANCY Job payloads must carry or deterministically derive proper tenant context. --- # 164. TENANT CONTEXT LOSS Scenario: ```text API has tenant A context ↓ job payload only contains resource ID ↓ worker queries globally ↓ wrong tenant resource matched ``` --- # 165. TENANT AUTH Workers do not inherit HTTP request headers or user session tokens. The service layer must explicitly enforce tenant boundaries. --- # 166. USER CONTEXT If a job executes on behalf of a user: verify: - user ID propagation - permission snapshots vs current permissions - handling of deactivated or revoked user accounts --- # 167. PERMISSION AT EXECUTION TIME If a user account is suspended before delayed execution: should the scheduled background job still proceed? This is a business and security decision. --- # 168. SYSTEM JOB Certain background maintenance runs under a privileged system principal. Document these explicitly. --- # 169. AUDIT For high-value background actions, document: - initiating principal - worker job ID - execution timestamp --- # 170. OBSERVABILITY For each critical queue, monitor: - queued count - active count - completed count - failed count - retried count - cancelled count - delayed count --- # 171. OLDEST AGE A crucial metric for tracking latency. --- # 172. PROCESSING DURATION Track P50, P95, and P99 latency where available. --- # 173. FAILURE RATE Track failure percentages per job type. --- # 174. RETRY RATE High retry frequencies often mask chronic downstream dependency failures. --- # 175. DEAD LETTER COUNT Must be exposed on operational dashboards. --- # 176. STUCK ACTIVE JOB Jobs remaining in active states for hours or days indicate orphaned workers or broken lease renewals. --- # 177. NOISE Expected business rejections should not trigger high-severity system alerts. --- # 178. LOG CONTEXT Job log entries must include: - job ID - logical operation ID - job type - attempt count - tenant and entity ID where safe --- # 179. SENSITIVE PAYLOAD LOG Never dump raw, unredacted job payloads into production logs. --- # 180. TRACE CONTEXT Propagate distributed trace context (OpenTelemetry/W3C) from HTTP intake into background jobs. --- # 181. TRACE SHOULD NOT SPAN 7 DAYS Asynchronous trace lifecycles should adapt appropriately to long delays rather than generating 7-day spans. --- # 182. METRICS CARDINALITY Do not use high-cardinality values like job IDs as metric labels. --- # 183. ALERTING Recommended production alerts: - oldest job age exceeding SLA - sustained error rate spikes - rapid queue depth accumulation - zero active consumers on queue - dead-letter queue growth --- # 184. HEALTH Worker process liveness alone: ```text process alive ``` does not prove workers are actively claiming and processing jobs. --- # 185. QUEUE CONNECTIVITY Readiness checks should reflect broker connectivity according to architectural standards. --- # 186. BROKER OUTAGE What happens to producers when the message broker is down? --- # 187. ENQUEUE FAILURE Critical HTTP operations must not return 200 OK if their required background tasks could not be durably enqueued. --- # 188. BROKER OUTAGE WORKER Workers must reconnect with backoff rather than entering aggressive busy loops. --- # 189. REDIS/BROKER RESTART Audit the persistence configuration (e.g., AOF vs RDB for Redis). --- # 190. QUEUE DURABILITY If the broker restarts: do pending and delayed jobs survive? --- # 191. EPHEMERAL QUEUE Acceptable for disposable analytics or telemetry. Unacceptable for financial or critical business processing without auxiliary persistence. --- # 192. BROKER DATA LOSS Verify configured broker durability guarantees, not default assumptions. --- # 193. BACKUP Backing up message brokers is rarely sufficient for disaster recovery. For critical operations, a transactional outbox in the primary database is far more resilient. --- # 194. REPLAY FROM SOURCE If the queue loses state, can pending jobs be reconstructed from authoritative database events? --- # 195. JOB HISTORY How long are completed and failed job records retained in broker memory? --- # 196. UNBOUNDED HISTORY Retaining millions of completed job records in Redis or database tables wastes storage and memory. --- # 197. AUTO-REMOVE Immediately purging completed jobs impedes post-incident debugging and audit verification. Balance based on criticality. --- # 198. FAILED HISTORY Failed job records should be retained longer for operational analysis. --- # 199. TESTS Map test coverage: - unit tests - worker integration tests - broker integration tests - retry logic tests - concurrency tests - crash-recovery tests - deployment versioning tests - backlog stress tests --- # 200. IN-MEMORY MOCK QUEUE A synchronous in-memory mock that invokes handlers immediately does not test true distributed queue semantics. --- # 201. REAL BROKER TEST Critical reliability workflows are best verified against realistic broker instances in test environments. --- # 202. DUPLICATE DELIVERY TEST Dispatch the identical job payload twice. --- # 203. CONCURRENT DUPLICATE TEST Dispatch the identical job concurrently across two worker processes. --- # 204. CRASH AFTER SIDE EFFECT TEST Execute the external side effect, then terminate the worker before acknowledgment. --- # 205. CRASH BEFORE SIDE EFFECT TEST Claim the job and terminate the process immediately. Verify the job is redelivered. --- # 206. VISIBILITY TIMEOUT TEST Execute a task whose duration intentionally exceeds the visibility lease timeout. --- # 207. RETRY TEST Simulate transient failures: ```text fail fail success ``` Verify attempt counters and backoff intervals. --- # 208. PERMANENT ERROR TEST Verify that invalid payloads do not trigger endless retry cycles. --- # 209. BROKER OUTAGE TEST Observe producer and consumer behavior during broker disconnects. --- # 210. DEPLOYMENT TEST Enqueue jobs with version 1 payloads, then deploy version 2 workers. --- # 211. STALE JOB TEST Mutate entity state between the enqueue moment and actual execution. --- # 212. CANCEL TEST Simulate cancellation while jobs are: - pending - delayed - active - completing --- # 213. BATCH PARTIAL FAILURE TEST Simulate a failure after partial records are processed. --- # 214. QUEUE FLOOD TEST Enqueue a sudden surge of jobs. Monitor: - queue depth - end-to-end latency - memory footprint - downstream database and API saturation --- # 215. FAIRNESS TEST Simulate one tenant generating a massive backlog. Verify whether other tenants continue receiving prompt execution. --- # 216. CRON DUPLICATION TEST Run multiple application replicas with embedded schedulers. --- # 217. CRON OVERLAP TEST Execute a scheduled task whose duration exceeds the recurrence period. --- # 218. DST TEST Simulate daylight saving time boundary shifts if schedules use local time zones. --- # 219. FINDING FORMAT Every significant finding must include: ```text ID: Severity: Category: Confidence: Status: Job: Queue: Trigger: Producer: Worker: Broker: Business resource: Tenant: File/Class: Function: Relevant config: Problem: Evidence: Job Timeline: T0: T1: T2: T3: Expected delivery semantics: Actual/Possible semantics: Durability point: Ack point: Retry behavior: Attempt: Concurrency: Side effect: Duplicate impact: Data impact: Financial impact: User impact: Backlog/capacity impact: Root cause: Recommended remediation: Regression/failure-injection test: Production verification: Complexity: XS / S / M / L / XL ``` --- # 220. SEVERITY Use: ## P0 - CRITICAL - duplicate job execution causes irreversible financial loss - worker tenant context failure causes cross-tenant corruption - background processing flaws cause catastrophic data corruption ## P1 - HIGH - critical jobs are permanently lost - standard retries or redeliveries duplicate critical side effects - deployments or restarts routinely leave jobs permanently stuck - queue backlogs cause severe business outages - critical scheduled jobs execute multiple times concurrently without protection ## P2 - MEDIUM - significant retry, concurrency, or stale-job defects - jobs experience severe latency or get stuck intermittently - worker capacity or tenant fairness defects with tangible operational impact ## P3 - LOW - minor edge-case anomalies - cosmetic logging or tracking defects ## P4 - IMPROVEMENT - observability, code structure, or tuning enhancements without confirmed business failure --- # 221. CONFIDENCE Use: ```text HIGH MEDIUM LOW ``` HIGH: source code, configuration, or tests directly confirm the scenario. MEDIUM: strong code evidence exists, but broker runtime configuration is partially unverified. LOW: depends on unverified third-party managed broker behaviors. --- # 222. STATUS Use: ```text CONFIRMED LIKELY THEORETICAL NOT VERIFIED ``` --- # 223. DELIVERY SEMANTICS STATUS Use: ```text AT-LEAST-ONCE AT-MOST-ONCE BEST-EFFORT UNKNOWN ``` Do not designate exactly-once without verified end-to-end domain safeguards. --- # 224. JOB IDEMPOTENCY STATUS Use: ```text NATURALLY IDEMPOTENT IDEMPOTENCY GUARDED DEDUP ONLY NOT IDEMPOTENT NOT VERIFIED ``` --- # 225. DURABILITY STATUS Use: ```text DURABLE EPHEMERAL PARTIALLY DURABLE NOT VERIFIED ``` --- # 226. FALSE-POSITIVE PREVENTION Before reporting P0/P1/P2 findings, verify: 1. producer logic 2. queue configuration 3. broker semantics 4. worker execution logic 5. acknowledgment sequence 6. retry policies 7. database constraints 8. side-effect implementations 9. deployment topology 10. test coverage Do not conclude a flaw exists based solely on an isolated worker handler. --- # 227. DO NOT ASSUME EXACTLY-ONCE Queue acknowledgment does not equal exactly-once business execution. --- # 228. DO NOT SOLVE DUPLICATES WITH DEDUP ALONE Deduplication does not protect against crashes occurring between side-effect execution and job acknowledgment. --- # 229. DO NOT INCREASE CONCURRENCY BLINDLY Backlogs can be caused by: - database bottlenecks - downstream 429 rate limits - lock contention - oversized jobs Spawning more workers can exacerbate system collapse. --- # 230. DO NOT DECREASE CONCURRENCY BLINDLY Can artificially restrict necessary throughput. --- # 231. DO NOT INTRODUCE QUEUES EVERYWHERE Brief, dependable synchronous operations often do not need queue complexity. --- # 232. DO NOT CONVERT BEST-EFFORT JOBS INTO CRITICAL Dropping an analytics event may be acceptable. Dropping payment settlement is not. --- # 233. DO NOT USE REDIS LOCKS AS THE DEFAULT ANSWER Conditional database writes, unique constraints, or idempotency keys are often cleaner and more reliable. --- # 234. DO NOT STORE ENTIRE ORM OBJECTS IN PAYLOADS Serialized entities often become stale by the time execution occurs. --- # 235. DO NOT MODIFY CODE During the audit: - do not alter queue parameters - do not modify retry configurations - do not change worker concurrency - do not insert outbox patterns - do not rewrite cron jobs - do not create dead-letter queues Complete the full investigation first. --- # 236. OUTPUT - BACKGROUND_JOBS_QUEUE_AUDIT.md Structure the final report: ## 1. Executive Summary - queue stack - broker infrastructure - job inventory overview - delivery semantics summary - top reliability vulnerabilities ## 2. Queue Architecture Map ## 3. Job Inventory ## 4. Job Criticality Classification ## 5. Producer / Enqueue Audit ## 6. Durability Audit ## 7. Payload / Schema Audit ## 8. Deployment Compatibility Audit ## 9. Worker Claim / Ack Audit ## 10. Idempotency / Duplicate Delivery Audit ## 11. Retry / Backoff Audit ## 12. Dead Letter / Poison Job Audit ## 13. Concurrency / Race Audit ## 14. Queue Backlog / Capacity Audit ## 15. Fairness / Multi-Tenant Audit ## 16. Scheduled / Cron Job Audit ## 17. Delayed / Stale Job Audit ## 18. Batch Job Audit ## 19. External API Worker Audit ## 20. Export / Import / Media Job Audit If applicable. ## 21. Cancellation Audit ## 22. Worker Shutdown / Deployment Audit ## 23. Broker Failure / Recovery Audit ## 24. Observability / Alerting ## 25. Test Coverage ## 26. Findings Summary | ID | Severity | Job | Category | Problem | Confidence | Status | |---|---|---|---|---|---|---| ## 27. P0 Findings ## 28. P1 Findings ## 29. P2 Findings ## 30. P3 Findings ## 31. P4 Improvements ## 32. Things Done Well ## 33. Unknown / Not Verified ## 34. Reliability Remediation Roadmap --- # 237. JOB MATRIX | Job | Queue | Criticality | Retry | Idempotent | Durable | |---|---|---|---|---|---| --- # 238. ACK MATRIX | Job | Side effect | Ack point | Crash window | Duplicate risk | |---|---|---|---|---| --- # 239. RETRY MATRIX | Error | Retryable | Backoff | Max attempts | Side-effect safe | |---|---|---|---|---| --- # 240. QUEUE CAPACITY MATRIX | Queue | Produce rate | Consume rate | Depth | Oldest age | Risk | |---|---:|---:|---:|---:|---| --- # 241. CRON MATRIX | Scheduled job | Frequency | Singleton | Reentrant | Missed-run policy | |---|---|---|---|---| --- # 242. SECOND PASS - CRASH AFTER SIDE EFFECT For each critical job, simulate: ```text side effect succeeds ↓ worker crashes ↓ ack not sent ↓ job redelivered ``` Determine what duplicates. --- # 243. SECOND PASS - CRASH BEFORE SIDE EFFECT ```text job claimed ↓ worker crashes ``` Verify whether the job becomes available for redelivery. --- # 244. SECOND PASS - VISIBILITY EXPIRY Execute a task whose runtime exceeds the visibility lease timeout. Determine whether another worker receives the same job concurrently. --- # 245. SECOND PASS - CONCURRENT DUPLICATE Execute two worker processes running the same logical task simultaneously. --- # 246. SECOND PASS - STALE JOB Mutate entity state after enqueue but before execution. Verify whether the worker re-checks required invariants. --- # 247. SECOND PASS - DEPLOYMENT Leave jobs pending in the queue. Deploy a new application release with updated payload or worker code. Verify whether older jobs can still be processed. --- # 248. SECOND PASS - QUEUE DOWN Attempt to enqueue jobs while the broker is down. Ask: > Does the caller receive success before the critical task is durable? --- # 249. SECOND PASS - WORKER DOWN The queue accumulates jobs without active consumers. Evaluate how quickly observability tooling detects this state. --- # 250. SECOND PASS - RETRY STORM A third-party API returns 503 for a large batch of jobs. Evaluate the volume of retries and backlog growth rate. --- # 251. SECOND PASS - 429 PROVIDER Verify whether concurrency and retries throttle or escalate load against rate-limited providers. --- # 252. SECOND PASS - POISON JOB Inject a permanently invalid payload. Verify: - retry limit - final destination - whether other tasks in the queue are blocked --- # 253. SECOND PASS - CRON MULTI-INSTANCE Launch multiple backend or scheduler replicas. Determine how many duplicate cron firings occur. --- # 254. SECOND PASS - CRON OVERLAP The job execution duration exceeds the recurrence interval. Verify whether executions overlap. --- # 255. SECOND PASS - BATCH CRASH A batch job reaches 70% completion before the worker crashes. Verify what happens upon retry. --- # 256. SECOND PASS - TENANT FLOOD A single tenant enqueues an enormous volume of jobs. Verify whether other tenants continue receiving prompt processing. --- # 257. SECOND PASS - CANCEL RACE ```text worker almost finished || user cancels ``` Determine which terminal state and side effects emerge. --- # 258. SECOND PASS - SHUTDOWN Send SIGTERM while workers have active jobs in flight. Track: - job claim behavior - acknowledgment handling - lease expiration - redelivery --- # 259. SECOND PASS - BROKER RESTART Verify: - pending jobs - delayed jobs - retry counters - active leases before and after a broker reboot. --- # 260. FINAL QUALITY GATE Before finalizing the audit report, confirm: - critical jobs have unambiguous durability guarantees - database commit and enqueue dual-write scenarios were evaluated - enqueueing before database transaction commit was investigated - payload snapshot vs current-state semantics were identified - forward and backward producer-worker compatibility was verified - acknowledgment points were mapped relative to external side effects - duplicate delivery was treated as an operational inevitability - deduplication was not conflated with true idempotency - crash-after-side-effect failure modes were analyzed - visibility timeout durations accommodate realistic job execution times - retry policies distinguish transient from permanent errors - nested retry cascades were investigated - poison jobs have a documented path to dead-letter storage - worker concurrency is calibrated against downstream capacity limits - queue backlogs are evaluated via depth and oldest job age - cron duplicate firings, misfires, and execution overlaps were checked - delayed jobs re-verify current domain invariants where appropriate - long-running batch jobs have partial failure and resume mechanisms - deployments do not abandon orphan queues or job types - production and staging queue namespaces are strictly isolated - worker execution preserves tenant boundaries securely - cancellation is not represented as an undo of irreversible side effects - broker outage behavior was explicitly evaluated - observability reliably surfaces missing consumers, stuck jobs, and DLQ growth - P4 tuning proposals are kept distinct from verified reliability bugs --- # FINAL RULE Do not deliver a report that merely states: > Add a queue, configure retries, use a dead-letter queue, and add more workers. That is not a background jobs audit. Look for real issues such as: ```text order DB transaction commits ↓ enqueue email/fulfillment job fails ↓ API returns success ↓ order exists ↓ required background processing never starts ``` or: ```text job sends payment/refund request ↓ provider succeeds ↓ worker crashes before queue ack ↓ job is redelivered ↓ same financial operation is attempted again ``` or: ```text visibility timeout = 30 s ↓ job normally runs 2 min ↓ worker A still processing ↓ queue makes job visible after 30 s ↓ worker B starts same job ↓ two executions overlap ``` or: ```text delayed job scheduled while resource ACTIVE ↓ resource becomes CANCELLED ↓ job executes 24 h later ↓ handler trusts stale payload ↓ cancelled resource receives forbidden side effect ``` or: ```text cron configured inside application process ↓ 5 backend replicas ↓ all five fire at midnight ↓ same billing job runs five times ``` or: ```text queue producer uses payload v1 ↓ deployment updates worker to v2 ↓ old v1 jobs remain pending ↓ new worker cannot deserialize them ↓ jobs permanently fail after deploy ``` or: ```text one tenant queues 500,000 exports ↓ shared FIFO queue ↓ other tenants' small jobs sit behind them ↓ system is technically alive ↓ legitimate jobs wait for hours ``` or: ```text worker performs side effect ↓ non-critical analytics call fails ↓ job throws ↓ queue retries entire operation ↓ critical side effect happens twice ``` These are the background job and queue defects you must uncover. Think through: - enqueue durability - acknowledgment timing - process crashes - duplicate deliveries - retries - stale state - concurrency - deployment version skew - queue backlogs - cron overlap - tenant fairness - downstream dependency limits For every critical finding, you must be able to answer: > When does the job become durable? > When does the queue consider the job complete? > What happens if the worker crashes before acknowledgment? > What happens if the worker crashes after executing a side effect? > Can the same logical job run concurrently on two workers? > Does retrying duplicate any operation? > Can pending jobs survive an application release? > Can a single tenant or job type starve the rest of the system? If broker delivery semantics are unverified: **QUEUE DELIVERY SEMANTICS NOT VERIFIED.** If an observation is simply an architectural enhancement without verified reliability risks: **P4 - IMPROVEMENT.** It is better to discover 6 real job-loss, duplicate execution, or stuck-task scenarios than to output 100 generic queue tips. The ultimate objective is a forensically rigorous Background Jobs & Queue audit that translates directly into: - crash-window regression tests - retry and idempotency hardening - stale-job safeguards - queue durability fixes - cron overlap protection - deployment compatibility adjustments - backlog and tenant fairness monitoring - production disaster recovery procedures
分类与标签
编程开发、后台任务、消息队列、幂等重试、可靠性审计、故障恢复