返回提示词目录第 1188 页

制定模型影子测试与灰度发布方案

为模型版本更新设计影子与灰度流程,产出路由代码、预设晋级指标、对比报告和回滚测试。

完整提示词

<context>
Offline evaluation does not catch everything that changes in production: feature pipelines that behave differently live, latency under real load, different input mixes, and effects that only show in business metrics. Shadow mode sends live requests to the new model without using its answers, which is safe only if the shadow path has no side effects (no writes, emails, charges or tool calls) and does not slow the user's request. Canary mode serves a small share of real users, which is needed to measure outcomes but exposes them to the new model; assignment should be sticky per user so experiences and metrics are not mixed. The usual failure is deciding the promotion rule after seeing the numbers. Labels often arrive late, so the early comparison is on agreement, prediction distribution, latency and errors, with outcome metrics later.
</context>

<task>
Plan and implement a safe rollout for: {{change}}

Serving: {{serving}}
{{#risk_appetite}}

Risk appetite: {{risk_appetite}}
{{/risk_appetite}}

1. Read the serving code and how the model version is chosen today. Identify every side effect a prediction triggers.
2. Choose the stages: shadow first (always, unless there is no live path), then canary at a small share, then ramp. Say why, and how long each stage runs given traffic volume.
3. Write promotion criteria before any data is seen: maximum latency increase at p95 and p99, maximum error rate, prediction distribution limits, agreement rate with the current model and how disagreements will be reviewed, and outcome or guardrail metrics for the canary with the minimum sample needed.
4. Implement shadow mode: asynchronous call to the new model off the request path, a timeout, no side effects (disable or stub them explicitly), and logging of both outputs with a shared request id and versions.
5. Implement canary routing: a configurable share, sticky assignment by user or account id, version recorded on every prediction, and a single switch to send all traffic back.
6. Build the comparison report: agreement rate, distribution of both models' outputs, the largest disagreements with examples, latency percentiles, errors, cost, and, for the canary, outcome metrics with confidence intervals.
7. Define rollback triggers that act automatically or page someone, and test the rollback switch.
8. Run the tests for routing, stickiness, side-effect isolation and rollback, and report real results.
</task>

<constraints>
- The shadow path must never change what the user receives or trigger side effects; prove it with a test.
- Do not change production traffic or configuration without explicit approval; deliver the code and plan.
- Promotion criteria are fixed before the test; do not relax them after seeing results without saying so.
{{> guardrails/scope-discipline}}
{{> guardrails/verify-before-done}}
</constraints>

<output_format>
## Rollout plan
Table: stage | traffic share | duration | exit condition.
## Promotion criteria
Numbered, each measurable.
## Code
Code blocks with file paths.
## Comparison report
The report template and how it is generated.
## Rollback
Triggers and the switch, with its test.
## Check
Commands run and real results.
</output_format>

可替换内容

  • {{change}}
  • {{serving}}
  • {{#risk_appetite}}
  • {{risk_appetite}}
  • {{/risk_appetite}}
  • {{> guardrails/scope-discipline}}
  • {{> guardrails/verify-before-done}}

分类与标签

编程开发、模型部署、影子测试、灰度发布、回滚验证

前往 AI 图片创作