返回提示词目录第 1344 页

分析并优化大模型成本与延迟

审视模型调用、令牌、检索和工具链路,设计可测量的优化实验并守住关键质量指标。

完整提示词

# LLM COST AND LATENCY OPTIMIZATION

I want a deep analysis of the cost and latency of the LLM system without degrading critical quality.

Main objective:

> Identify where tokens, model calls, retrieval, tool calls and waiting are spent unnecessarily, and then prove the optimization through measurable before/after results with a quality regression guard.

This is not:

- "use a cheaper model"
- "shorten the prompt"
- "reduce max_tokens"
- "add a cache"
- automatic routing to a small model
- optimizing the average while p95/p99 stays bad
- savings without a quality eval

Priority:

**critical quality preserved > correctness > reliability > cost > p95 latency > average latency**

## 1. BASELINE

Measure:

```text
requests/day
model calls/request
input tokens
cached input
output tokens
tool calls
retrieval calls
latency
cost/request
cost/day
```

## 2. SEGMENT

By use case.

## 3. MODEL

## 4. INPUT SIZE

## 5. OUTPUT SIZE

## 6. TOOL PATH

## 7. SUCCESS/FAILURE

## 8. RETRY

## 9. FALLBACK

## 10. STATIC PROMPT TOKENS

## 11. DYNAMIC CONTEXT TOKENS

## 12. CONVERSATION HISTORY

## 13. RAG TOKENS

## 14. TOOL OUTPUT TOKENS

## 15. DUPLICATE CONTEXT

## 16. REDUNDANT SYSTEM RULE

## 17. FULL DOCUMENT

Could be narrowed.

## 18. HISTORY COMPACTION

Must preserve facts.

## 19. SUMMARY DRIFT

## 20. CACHE

Prompt/provider caching if available.

## 21. APPLICATION CACHE

## 22. SEMANTIC CACHE

Risk.

## 23. USER SCOPE

## 24. FRESHNESS

## 25. CACHE HIT RATE

## 26. MODEL ROUTING

## 27. TASK COMPLEXITY

## 28. SMALL MODEL

Evaluate.

## 29. LARGE MODEL

Use only where benefit proven.

## 30. CASCADE

Small first, large on uncertainty.

## 31. CASCADE FALSE NEGATIVE

Small model incorrectly thinks it succeeded.

## 32. ROUTER

Needs eval.

## 33. CONFIDENCE

Do not rely on model self-confidence alone.

## 34. OUTPUT TOKENS

## 35. STOP

## 36. STRUCTURED OUTPUT

Can reduce verbosity.

## 37. STREAMING

Improves perceived latency, not total compute.

## 38. TTFT

## 39. TOKENS/SECOND

## 40. TOTAL LATENCY

## 41. RETRIEVAL LATENCY

## 42. RERANK LATENCY

## 43. TOOL LATENCY

## 44. SERIAL CALLS

## 45. PARALLEL CALLS

Only independent calls.

## 46. SPECULATIVE PARALLELISM

Can waste cost.

## 47. DEPENDENCY GRAPH

## 48. N+1 MODEL CALL

## 49. LOOP

## 50. MULTI-AGENT

Can multiply cost.

## 51. JUDGE CALL

## 52. SELF-CRITIQUE CALL

Measure benefit.

## 53. RETRY COST

## 54. INVALID JSON RETRY

Improve schema/prompt.

## 55. TOOL ERROR RETRY

## 56. PROVIDER RATE LIMIT

## 57. CONCURRENCY

## 58. BATCH

Embeddings/classification where provider supports.

## 59. ASYNC

Non-interactive task.

## 60. BACKGROUND

## 61. PRIORITY QUEUE

## 62. SLA

Not all tasks need same latency.

## 63. TIMEOUT

## 64. DEADLINE PROPAGATION

## 65. BUDGET

Per task/user/tenant.

## 66. MAX STEPS

## 67. TOKEN CAP

## 68. COST CAP

## 69. PROVIDER PRICING

Must use current verified pricing if calculating real money.

If not verified:

**CURRENT PROVIDER PRICING: NOT VERIFIED**

## 70. CACHED TOKEN PRICING

Provider-specific.

## 71. TOOL/API COST

## 72. VECTOR DB COST

## 73. STORAGE/LOG COST

## 74. UNIT ECONOMICS

Cost per successful task.

## 75. COST PER BUSINESS OUTCOME

More useful than cost per request where possible.

## 76. FAILURE COST

Failed task still costs money.

## 77. REWORK COST

Bad AI answer causes human work.

## 78. HUMAN ESCALATION

May be cheaper than huge model loop.

## 79. QUALITY BASELINE

Mandatory.

## 80. EVAL SET

## 81. QUALITY BY SEGMENT

## 82. CRITICAL CASE

## 83. NON-CRITICAL CASE

## 84. OPTIMIZATION EXPERIMENT

One variable at time where possible.

## 85. BEFORE/AFTER

## 86. STATISTICAL VARIANCE

Repeated runs.

## 87. COST SAVINGS

## 88. LATENCY SAVINGS

## 89. QUALITY DELTA

## 90. CRITICAL REGRESSION

Blocks rollout.

## 91. CANARY

## 92. ROLLBACK

## 93. OBSERVABILITY

Per request:

- model
- input/output token
- cache hit
- cost
- latency
- tools
- retries
- outcome

## 94. OUTLIER

## 95. EXPENSIVE USER/TENANT

## 96. ABUSE

## 97. RATE LIMIT

## 98. COST ANOMALY

## 99. DAILY BUDGET

## 100. FALSE POSITIVE RULES

Do not report:

- large model
- long prompt
- multiple model calls
- no cache
- streaming
- retrieval

as a problem without workload evidence.

## 101. EVIDENCE TIERS

```text
A - measured production/controlled benchmark
B - reproducible profiling
C - strong static cost path
D - hypothesis
E - optimization idea
```

## 102. STATUS

```text
CONFIRMED
LIKELY
NOT VERIFIED
CONTROLLED
NOT APPLICABLE
HARDENING
```

## 103. SEVERITY

P0:
cost runaway threatens system/account availability or creates catastrophic financial exposure

P1:
repeatable high-volume runaway or severe latency failure in critical path

P2:
material cost/latency inefficiency

P3:
limited optimization

P4:
optional tuning

## 104. FINDING FORMAT

```text
ID:
Severity:
Status:
Evidence tier:
Flow:
Model:
Volume:
Current tokens:
Current calls:
Current latency:
Current cost:
Cause:
Optimization:
New latency:
New cost:
Quality delta:
Critical regression:
Evidence:
Rollout:
```

## 105. COST MATRIX

| Flow | Calls | In tokens | Out tokens | Cost | Success |
|---|---:|---:|---:|---:|---:|

## 106. LATENCY MATRIX

| Stage | p50 | p95 | p99 | Dependency |
|---|---:|---:|---:|---|

## 107. MODEL ROUTING MATRIX

| Task | Current | Candidate | Quality | Cost | Latency |
|---|---|---|---:|---:|---:|

## 108. SECOND PASS

Test:

- long conversation
- max document size
- fallback model
- provider retry
- no cache
- cache hit
- 10x concurrency
- expensive tenant
- tool timeout
- agent loop
- small-model route
- quality-critical cases

## 109. FINAL QUALITY GATE

Confirm:

- baseline
- segmentation
- model calls
- tokens
- cache
- retrieval
- tools
- retries
- routing
- concurrency
- p95/p99
- provider pricing verification
- quality eval
- rollout guard
- observability

## 110. OUTPUT

`LLM_COST_LATENCY_OPTIMIZATION.md`

## 111. FAILURE CHAINS

```text
every user turn sends entire 200-message history
↓
static context grows continuously
↓
input tokens dominate cost
↓
latency and price rise with session age
↓
no measured benefit from older messages
```

```text
cheap model classifies task
↓
false "easy" classification routes complex task to weak model
↓
output passes schema
↓
quality silently drops
↓
cost improves but business outcome worsens
```

# FINAL RULE

The cheapest AI system is not the one with the smallest token bill.

The best optimization target is:

```text
cost per successful, correct, useful task
```

with the critical quality gates preserved.

分类与标签

编程开发、大模型系统、成本优化、延迟分析

前往 AI 图片创作