机器学习实验方案规划
围绕预测目标与数据集制定实验计划,明确评估指标、基线、数据切分、泄漏检查、运行步骤和停止条件。
完整提示词
<context>
Weeks of modelling are lost to the same mistakes. With no baseline, "0.92 AUC" means nothing. Random splits on data with time or group structure leak the answer into training. Features computed after the moment of prediction make offline results impossible to reproduce in production. The chosen metric does not match the decision the model supports. Tuning against the test set inflates every number. Without a stop rule, the project drifts from run to run. All of this is cheapest to fix on paper, before training code exists.
</context>
<task>
Plan an experiment for this problem:
{{problem}}
Dataset:
{{dataset}}
{{#compute_budget}}
Compute budget: {{compute_budget}}
{{/compute_budget}}
1. Frame it: the target, the unit of prediction (a row, user, session, document), the moment of prediction and which features exist at that moment, the decision the output drives, and the cost of a false positive against a false negative. If the target or the moment of prediction is unclear, ask before planning further.
2. Choose metrics: one primary metric that matches the decision (for example recall at a fixed precision for rare positives, PR-AUC for imbalanced ranking, MAE in the target's units), guardrail metrics, the slices to report separately, and the smallest improvement that would change the decision.
3. Define baselines in order: a trivial one (majority class, mean, last value, seasonal naive), a heuristic a domain expert would write, and a simple model such as logistic regression or gradient-boosted trees on obvious features. Every later result is reported against all three.
4. Design the splits: by time when the model will predict the future, by group when the same user, patient or document appears in many rows, stratified when classes are rare, cross-validated when data is small. Lock the test set until the final evaluation.
5. List leakage checks specific to this dataset: features recorded after the moment of prediction, identifiers or timestamps that correlate with the label, duplicates or near-duplicates across splits, preprocessing fitted on all the data, and target encoding computed outside the training fold. For each, give the concrete check, and treat a result that looks too good as a leak until proven otherwise.
6. Write the run plan: ordered runs, each with a hypothesis, the single change, its expected effect, its compute cost, and the evidence that would confirm it. Include ablations that attribute any gain over the simple model, and at least three seeds wherever variance could exceed the gain.
7. Specify reproducibility: data snapshot or version, code commit, configuration and seeds recorded for every run.
8. Write the stop rule: the condition to stop (target met, budget spent, or no gain above the minimum over a set number of consecutive runs) and the result that would end the project.
</task>
<constraints>
- Do not write training code. This is the plan the code will follow.
- Fit the run plan inside the compute budget, and say what to drop if it does not fit.
- Prefer the simplest model that meets the decision's needs. A complex model must beat the simple one by more than seed variance to stay in the plan.
{{> output/uncertainty}}
</constraints>
<output_format>
## Framing
Target, unit, moment of prediction, decision, error costs.
## Metrics
Primary, guardrails, slices, minimum meaningful improvement.
## Baselines
The three baselines and how each is computed.
## Data splits
The split scheme and why it matches how the model will be used.
## Leakage checks
Checklist: suspected leak, check, action if found.
## Run plan
Table: # | hypothesis | change | cost | what confirms it.
## Reproducibility
What is recorded for every run, and where.
## Stop rule
When to stop, and what would end the project.
</output_format>可替换内容
- {{problem}}
- {{dataset}}
- {{#compute_budget}}
- {{compute_budget}}
- {{/compute_budget}}
- {{> output/uncertainty}}
分类与标签
编程开发、机器学习实验、模型评估、数据切分