将训练模型部署为推理接口
复用训练预处理搭建模型服务,提供校验、版本与健康检查,并交付一致性测试和压测结果。
完整提示词
<context>
The common failures of model APIs are not in the model. Preprocessing re-implemented in the service drifts from training, so predictions are wrong with no error (training-serving skew); the same code or the saved pipeline must be used in both. The model is loaded per request, or lazily on the first one, which then times out. CPU-heavy inference inside an async handler blocks the event loop and stalls every other request; it belongs in a thread or process pool, or a sync worker model. GPUs sit mostly idle without batching requests together. Missing input validation lets out-of-range values, missing features or huge payloads through. Responses without a model version make incidents impossible to trace. A readiness check that returns OK before the model has loaded sends traffic to a cold instance. And nobody knows the real capacity until a load test measures p95 and p99 latency.
</context>
<task>
Serve this model: {{model}}
Traffic: {{traffic}}
{{#platform}}
Platform: {{platform}}
{{/platform}}
1. Read the training code, the saved artifacts and any existing service code. Reuse the existing web framework and deployment pattern; FastAPI is the default only if nothing exists.
2. Make preprocessing shared: load the saved pipeline, or import the same preprocessing module used in training. Do not re-implement transformations in the service.
3. Define request and response schemas: field types, ranges, allowed categories, maximum batch size and payload size; responses carry predictions, the model version and a request id.
4. Load the model once at startup, warm it with a sample prediction, and only then report ready. Provide separate liveness and readiness endpoints.
5. Run inference without blocking the server: a thread or process pool for CPU-bound models, or dynamic micro-batching with a short wait window for GPU models. Set request timeouts.
6. Version the model: read the version from the artifact's metadata, expose it, and allow loading a specific version by configuration.
7. Log per request: latency, model version, input validation failures and prediction summaries for monitoring, without logging personal data.
8. Write tests: valid request, each validation failure, batch request, readiness before and after load, and a parity test that the API output equals the offline model output on a fixed sample.
9. Load test at the expected rate and twice it, reporting p50, p95 and p99 latency, error rate and resource use, and state the instance count needed.
</task>
<constraints>
- Never let the service's preprocessing differ from training; the parity test proves it.
- Do not log raw inputs that may hold personal data.
- Do not run load tests against shared or production environments without asking.
{{> guardrails/scope-discipline}}
{{> guardrails/verify-before-done}}
</constraints>
<output_format>
## Design
Bullets: framework, preprocessing reuse, concurrency model, batching, versioning, health checks.
## Code
Code blocks with file paths.
## Tests
Code blocks with file paths.
## Load test
Table: rate | p50 | p95 | p99 | error rate | CPU or GPU use.
## Check
Commands run and real results.
## Open questions
Numbered, or "None".
</output_format>可替换内容
- {{model}}
- {{traffic}}
- {{#platform}}
- {{platform}}
- {{/platform}}
- {{> guardrails/scope-discipline}}
- {{> guardrails/verify-before-done}}
分类与标签
编程开发、模型部署、推理接口、训练推理一致性、服务压测