模型微调可行性与实施规划
比较提示优化、检索与微调路径,制定数据、训练、评估和成本方案,明确上线及回滚条件。
完整提示词
<context>
Fine-tuning changes how a model behaves: output format and style, consistency on a narrow classification or extraction task, reliability at calling tools, or a large model's skill distilled into a smaller, cheaper one. It is a poor way to teach facts that change, which retrieval handles better, and it cannot fix a task nobody has specified clearly. Most fine-tuning projects that fail never measured a strong prompted baseline, trained on noisy or leaky data, or forgot the recurring costs: relabelling, retraining when the base model is retired, and hosting.
</context>
<task>
Task:
{{task}}
Data available:
{{data_available}}
{{#budget}}
Budget: {{budget}}
{{/budget}}
1. Compare the options for this task: a better prompt with few-shot examples and structured output, retrieval, supervised fine-tuning, preference tuning (only if pairwise preferences exist or can be collected), and distillation from a larger model. Judge each against what is failing now, the data's volume and quality, how often the task changes, request volume, latency, and whether a small or self-hosted model is required.
2. Give a verdict: do not fine-tune, fine-tune after a baseline, or fine-tune now. If no prompted baseline has been measured, the first step is always to build the eval set and the best prompt baseline, and to set the lift fine-tuning must achieve to be worth it.
3. If fine-tuning stays on the table, plan the data:
- the format: chat-style JSONL with the same system prompt used at inference, and tool calls included if the task uses tools;
- how to build examples from the data available, and how many are needed, stated as rules of thumb (format or style tasks often need tens to a few hundred good examples; classification over many labels needs more per label);
- cleaning: deduplication, label consistency checks, removal of personal data;
- splits: train, validation and a locked test set, split by source, customer or time so near-duplicates do not cross splits.
4. Plan training: full fine-tune, adapter methods such as LoRA, or a hosted fine-tuning API, and why. Give starting settings (epochs, learning rate or the platform's multiplier, batch size), the signals to watch (validation loss rising while training loss falls means overfitting), and a sweep of at most three runs.
5. Plan evaluation: the same eval set for the base model, the prompted baseline and each fine-tuned run; per-slice results; checks that general behaviours the product relies on (refusals, format, tone) did not regress; and a human review sample.
6. Model cost as formulas, filling in only numbers the user gave: labelling hours, training tokens (examples × average tokens × epochs × price per token), the inference price difference times monthly volume, hosting, and retraining frequency. Give the break-even volume.
7. State go/no-go criteria and how to roll back.
</task>
<constraints>
- Never invent prices or benchmark results. Use variables where the user gave no figure.
- Keep the plan vendor-neutral. Name a platform only as an example.
- If the budget cannot cover the plan, say what to cut first.
- Do not recommend fine-tuning to inject knowledge that changes more often than you would retrain.
{{> output/uncertainty}}
</constraints>
<output_format>
## Verdict
One line, then two or three sentences of reasoning.
## Why
Table: approach | fit for this task | cost | main risk.
## Baseline first
The prompt baseline to build, the eval set, and the target lift.
## Data plan
Format, sources, cleaning, splits and target size.
## Training plan
Method, starting settings, runs and what to watch.
## Evaluation
What is compared, on which slices, and what counts as a win.
## Cost model
One-off and recurring costs as formulas, with break-even volume.
## Go/no-go
The criteria to ship, and the rollback.
</output_format>可替换内容
- {{task}}
- {{data_available}}
- {{#budget}}
- {{budget}}
- {{/budget}}
- {{> output/uncertainty}}
分类与标签
编程开发、模型微调、训练数据、模型评估、成本测算、上线回滚