返回提示词目录第 1348 页

人工智能模型选型与评测

针对产品真实任务设计模型评测,比较质量、延迟、成本与隐私,输出选型和路由方案。

完整提示词

# AI MODEL SELECTION AND EVALUATION

I want a rigorous, evidence-first process for selecting an AI model for a specific product or workflow.

Main objective:

> Choose a model based on the real task distribution, quality, reliability, latency, tool support, context behavior, privacy/security requirements and cost, without benchmark marketing and without a universal "best model".

This is not:

- ranking models by general impression
- blindly following a leaderboard
- choosing the newest model
- choosing the most expensive model
- choosing the cheapest model
- one benchmark for all tasks
- relying on a vendor demo
- testing only English if the product is not English-only

## 1. DEFINE TASK DISTRIBUTION

Inventory the real tasks.

For each:

```text
Task:
Frequency:
Criticality:
Input type:
Context size:
Output:
Tools:
Freshness:
Latency target:
Cost sensitivity:
Languages:
```

## 2. HARD REQUIREMENTS

Before the quality eval, filter by:

- modalities
- context
- tools
- structured output
- region
- privacy
- data retention
- throughput
- API availability

## 3. CURRENT PROVIDER FACTS

If features, pricing or limits may have changed:

verify current official information.

If not:

**CURRENT MODEL CAPABILITY/PRICING: NOT VERIFIED**

## 4. CANDIDATE SET

Do not include a model only because it is popular.

## 5. BASELINE

Current model or human/process baseline.

## 6. DATASET

Representative.

## 7. PRODUCTION DISTRIBUTION

## 8. EDGE CASE

## 9. HARD CASE

## 10. EASY CASE

## 11. UNANSWERABLE

## 12. ADVERSARIAL

## 13. LONG CONTEXT

## 14. SHORT CONTEXT

## 15. MULTI-LANGUAGE

## 16. DOMAIN

## 17. TOOL TASK

## 18. STRUCTURED OUTPUT

## 19. EXTRACTION

## 20. SUMMARIZATION

## 21. REASONING

## 22. CODE

## 23. RAG

## 24. AGENT

## 25. MULTIMODAL

## 26. REPETITIONS

## 27. NONDETERMINISM

## 28. PARAMETERS

Keep fair/configured.

## 29. PROMPT ADAPTATION

One shared prompt may unfairly favor model.

Evaluate:

- same prompt baseline
- then reasonable per-model optimization

Document both.

## 30. QUALITY RUBRIC

Define observable criteria.

## 31. BINARY CORRECTNESS

Where objective.

## 32. HUMAN RUBRIC

## 33. LLM JUDGE

## 34. JUDGE BIAS

## 35. BLIND EVAL

Hide model identity from humans if practical.

## 36. PAIRWISE

## 37. INTER-RATER

## 38. CRITICAL ERROR

Weight.

## 39. AVERAGE SCORE

Can hide catastrophic failures.

## 40. PASS@K

Where repeated attempts matter.

## 41. FIRST-TRY

Important for user-facing.

## 42. TOOL ACCURACY

## 43. ARGUMENT ACCURACY

## 44. TOOL COMPLETION

## 45. SCHEMA COMPLIANCE

## 46. HALLUCINATION

## 47. GROUNDING

## 48. REFUSAL

## 49. OVER-REFUSAL

## 50. INSTRUCTION FOLLOWING

## 51. LONG-CONTEXT RETRIEVAL

## 52. POSITION SENSITIVITY

## 53. LATENCY

- TTFT
- total
- p95

## 54. THROUGHPUT

## 55. RATE LIMIT

## 56. CONCURRENCY

## 57. STABILITY

## 58. PROVIDER OUTAGE

Historical/provider evidence if available.

## 59. FALLBACK

## 60. COST

Use current verified pricing.

## 61. INPUT TOKENS

## 62. OUTPUT TOKENS

## 63. CACHE

## 64. TOOL COST

## 65. COST PER PASS

## 66. COST PER SUCCESSFUL TASK

## 67. HUMAN REWORK

## 68. ROUTING

Maybe multiple models outperform one universal model.

## 69. EASY/HARD ROUTING

## 70. ROUTER ERROR

## 71. CASCADE

## 72. FALLBACK MODEL

## 73. PROVIDER DIVERSITY

## 74. OPERATIONAL COMPLEXITY

Two providers add complexity.

## 75. PRIVACY

## 76. RETENTION

## 77. REGION

## 78. COMPLIANCE

If relevant, verify actual requirements.

## 79. MODEL VERSIONING

Pinned vs alias.

## 80. SILENT PROVIDER UPDATE

If alias behavior changes.

## 81. DEPRECATION

## 82. MIGRATION COST

## 83. PROMPT PORTABILITY

## 84. TOOL PORTABILITY

## 85. OUTPUT DIFFERENCE

## 86. EVAL REPRODUCIBILITY

Store:

```text
model ID
date
prompt version
dataset version
parameters
results
```

## 87. CURRENTNESS

Re-run periodically.

## 88. DRIFT

Provider/model behavior changes.

## 89. CANARY

## 90. SHADOW

## 91. A/B

Where safe.

## 92. USER SEGMENT

## 93. LANGUAGE SEGMENT

## 94. TASK SEGMENT

## 95. OUTCOME METRIC

## 96. COST GUARD

## 97. QUALITY GUARD

## 98. ROLLBACK

## 99. MODEL ROUTING ARCHITECTURE

If multiple.

## 100. FALSE POSITIVE RULES

Do not conclude:

- bigger model wins
- latest model wins
- benchmark leader wins
- cheaper model is better
- faster model is better
- open model is better/worse
- proprietary model is better/worse

without task evidence.

## 101. EVIDENCE TIERS

```text
A - controlled task-specific eval / production experiment
B - reproducible benchmark on representative data
C - official capability/runtime facts
D - inference from external benchmarks
E - anecdote/vendor claim
```

A vendor claim is not enough for a selection conclusion.

## 102. STATUS

```text
VERIFIED
LIKELY
NOT VERIFIED
NOT APPLICABLE
```

## 103. SEVERITY

Severity applies to risks in the current model choice or in the evaluation process, not to ranking models.

P0:
- the selected model or routing automatically performs a critical/high-stakes task with unverified quality and causes harmful actions at scale
- the selection violates a hard requirement for sensitive data (privacy, region, retention)

P1:
- the selected model fails a hard requirement (tools, context, structured output, throughput) or makes repeatable critical errors in a critical task class that the evaluation did not cover
- no fallback or rollback exists for a model already in production

P2:
- a significant regression in quality, latency or cost per successful task on an important segment
- the evaluation is not representative of the real task distribution

P3:
- a limited gap: a missing segment, too few repetitions, undocumented parameters

P4:
- process hardening: periodic re-evaluation, shadow testing, drift monitoring

Report a risk as confirmed only with evidence tier A or B.

## 104. IMPORTANT DECISION RULE

Do not build one global numeric score unless the weights are explicitly business-defined.

Show the trade-off instead.

## 105. MODEL COMPARISON MATRIX

| Model | Critical quality | General quality | p95 | Cost/success | Tools |
|---|---:|---:|---:|---:|---|

## 106. TASK MATRIX

| Task | Model A | Model B | Model C | Notes |
|---|---:|---:|---:|---|

## 107. FAILURE MATRIX

| Model | Critical failures | Refusal | Schema fail | Tool fail |
|---|---:|---:|---:|---:|

## 108. ROUTING MATRIX

| Task class | Primary | Fallback | Escalation condition |
|---|---|---|---|

## 109. DECISION OUTPUT

Do not give only:

```text
Model X is best.
```

Show:

```text
Task A:
observed strengths/weaknesses

Task B:
observed strengths/weaknesses

Tradeoffs:
quality
cost
latency
operations
```

If the user asks for a final selection for a non-political product decision, you can give a recommendation based on clearly documented weights.

## 110. SECOND PASS

Repeat the evaluation:

- hard cases only
- long context
- tool tasks
- unanswerable
- non-English
- provider retry/failure
- fallback
- high concurrency
- current pricing
- critical error analysis

## 111. FINAL QUALITY GATE

Confirm:

- real task distribution
- requirements
- current capability verification
- representative dataset
- repeated runs
- critical error weighting
- tools
- grounding
- refusal
- languages
- context
- latency
- throughput
- cost
- privacy
- operational complexity
- routing
- rollout
- drift

## 112. OUTPUT

`AI_MODEL_SELECTION_EVALUATION.md`

## 113. FAILURE CHAINS

```text
public benchmark shows Model A ahead
↓
product mostly processes Serbian legal documents
↓
benchmark is English general reasoning
↓
no local-language evaluation exists
↓
team migrates
↓
critical extraction quality declines
```

```text
Model B costs 60% less per token
↓
requires twice as many retries
↓
outputs are longer
↓
human correction rises
↓
cost per successful task is higher
```

```text
Model C wins single-turn benchmark
↓
production task requires 12-step tool workflow
↓
tool argument error rate is much higher
↓
end-to-end completion is lower despite stronger raw reasoning
```

# FINAL RULE

Do not look for the universally best model.

Look for:

```text
the best model
for a specific task distribution
with specific quality thresholds
in a specific latency/cost/privacy environment
```

and repeat the evaluation when the models, pricing, provider behavior or the product itself change significantly.

分类与标签

编程开发、模型选型、性能评测、成本比较、模型路由、生产部署

前往 AI 图片创作