返回提示词目录第 1365 页

为外部调用配置重试与熔断

帮助检查调用链与幂等性,制定超时、退避和熔断策略,并补充可验证的实现与测试。

完整提示词

<context>
Retry code often makes outages worse: no timeout at all, so threads wait forever on a hung connection; retries on every error, including 400s and validation failures that will never succeed; retrying a payment or order creation without an idempotency key, so the customer is charged twice; fixed delays that make every client retry in lockstep; retries at three layers multiplying into dozens of attempts per user request; total retry time longer than the caller's own deadline; and no circuit breaker, so a dead dependency ties up every worker. Good resilience is a written policy per call: how long to wait, what to retry, how often, how to stay safe to repeat, and when to stop trying for a while.
</context>

<task>
Add timeouts, retries and circuit breaking to this call site:

<call_site>
{{call_site}}
</call_site>

Stack: {{stack}}
{{#slo}}
Caller's budget: {{slo}}
{{/slo}}

1. Read the call site and everything around it: the client and its current settings, every caller and their own deadlines, existing retry logic at other layers (client libraries, service mesh, load balancer, job queue), whether the operation is idempotent, whether the provider supports idempotency keys, and the provider's documented rate limits and error codes. Describe current behaviour before changing it. If you cannot tell whether the operation is safe to repeat, ask and stop.
2. Write the policy as a table:
   - Timeouts: a connect timeout, a per-attempt timeout, and an overall deadline that fits inside the caller's budget and propagates the caller's cancellation. Derive the numbers from the budget, or propose them from observed latency and mark them as proposed.
   - Retry conditions: only transient failures, such as connection errors, timeouts on idempotent calls, HTTP 502, 503 and 504, and 429 honouring `Retry-After`; for gRPC, `UNAVAILABLE`, `RESOURCE_EXHAUSTED` with backoff, and `DEADLINE_EXCEEDED` only on idempotent calls. Never retry 400, 401, 403, 404, 409 or 422 (or `INVALID_ARGUMENT`, `PERMISSION_DENIED`, `NOT_FOUND`, `ALREADY_EXISTS`, `FAILED_PRECONDITION`), or errors the provider marks permanent.
   - Backoff: exponential with full jitter, a cap on each delay, a maximum number of attempts, and a total retry budget that ends before the overall deadline.
   - Idempotency: reads retry freely; writes retry only with an idempotency key generated once per logical operation and reused on every attempt, or when the operation is naturally idempotent.
   - Circuit breaker: opens on a failure rate over a sliding window with a minimum number of calls, stays open for a cool-down, half-opens with limited trial calls, and has a defined fallback when open (cached value, degraded response, queued for later, or a clear error).
   - Concurrency: a bulkhead limit, if a slow dependency could exhaust shared workers.
3. Implement it with the resilience library or client features the project already uses, or a small well-tested helper if none exists, configured from settings rather than hard-coded. Retry at one layer only, and remove or disable retries at other layers if they would multiply.
4. Add observability: metrics for attempts, retries, timeouts and breaker state changes; a log line per final failure with the attempt count and the last error, without request bodies or secrets; and propagate the trace context.
5. Write tests with a fake server or stubbed transport and a controllable clock and random source: a hung response hits the per-attempt timeout; a transient error followed by success retries and succeeds; 4xx responses are not retried; `Retry-After` is honoured; attempts stop when the budget runs out; the same idempotency key is sent on every attempt; the breaker opens after the threshold, rejects fast while open and closes after successful trial calls; and jittered delays stay within bounds. Run them and report the real result.
</task>

<constraints>
- Never add retries to a non-idempotent write without an idempotency mechanism; if none exists, say so and stop at timeouts and the circuit breaker.
- Total time including retries must fit inside the caller's deadline.
- Do not change the external contract of the call (return types, error types callers depend on) without listing every caller affected.
{{> guardrails/scope-discipline}}
{{> guardrails/verify-before-done}}
{{> guardrails/no-hardcoding-to-pass-tests}}
</constraints>

<output_format>
## Current behaviour
Timeouts, retries and failure handling as they are today, and any retries at other layers.
## Policy
Table: setting, value, reason. Proposed values are marked proposed.
## Changes
One line per file.
## Tests
One line per test and the real result of the run.
## Rollout and monitoring
How to roll it out safely, which metrics and alerts to watch, and how to tune the values.
</output_format>

可替换内容

  • {{call_site}}
  • {{stack}}
  • {{#slo}}
  • {{slo}}
  • {{/slo}}
  • {{> guardrails/scope-discipline}}
  • {{> guardrails/verify-before-done}}
  • {{> guardrails/no-hardcoding-to-pass-tests}}

分类与标签

编程开发、接口调用、故障恢复、服务韧性

前往 AI 图片创作