排查模型训练代码数据泄漏
追踪训练数据从加载到评估的流向,给出泄漏证据、影响程度、最小修复方案与确认测试。
完整提示词
<context>
When a model scores far better offline than in production, leakage is the first suspect, and it usually hides in ordinary-looking lines. Scalers, imputers, PCA, feature selection, tokenisers or oversampling (SMOTE) fitted before the split. Target or mean encoding computed on the full table. Random splits on time-ordered data, rolling windows without a shift, backward fills that copy future values into the past. Rows of the same customer, patient or document in both train and test. Columns recorded after the outcome (status fields, closing dates, "reason" fields). The test set used for early stopping, threshold choice or model selection. Exact and near-duplicate rows across splits. Each of these can be confirmed with a cheap test, and a review that only lists suspicions without a test leaves the team arguing.
</context>
<task>
Review this training code for leakage: {{code}}
Context: {{context}}
1. Read the code end to end in execution order, following data from loading to the reported metric. Note where the split happens; everything fitted before that line is suspect.
2. Check each leakage class against the code and the stated moment of prediction:
- preprocessing or resampling fitted before the split or outside cross-validation folds;
- target-derived features and encodings computed on the full data;
- features available only after the moment of prediction (check column names, joins on later tables, timestamps);
- time leakage: random splits on temporal data, unshifted rolling or lag features, forward or backward fills across the split, future-dated joins;
- group leakage: the same entity in more than one split;
- duplicates and near-duplicates across splits;
- evaluation leakage: test data used for early stopping, tuning, threshold or model selection, or the test set reused across many experiments;
- for text and images: vocabulary, normalisation statistics or augmentation fitted on all data, and the same source document or scene in several splits.
3. For each finding give the file and line, why it leaks, the severity (inflates the headline metric a lot, a little, or only in theory), and the minimal fix.
4. Propose a confirmation test for each high-severity finding: retrain with the fix and compare, count shared entities or duplicate hashes across splits, drop the suspect feature and watch the score, or run adversarial validation (a classifier that tries to tell train from test).
5. If the data is available and the user allows running code, run the cheap checks (duplicate and entity overlap counts, feature-drop comparison) and report the real numbers.
6. List the areas you checked and found clean, so the team knows what was covered.
</task>
<constraints>
- Every finding cites the exact code; do not report a leak you cannot point to. Mark suspicions that depend on data you could not see as such.
- Do not rewrite the training code in this review; give the minimal fix per finding.
- Do not run long training jobs without asking; the cheap checks come first.
{{> guardrails/investigate-before-answering}}
</constraints>
<output_format>
## Verdict
One paragraph: how likely the reported score is inflated, and by which finding most.
## Findings
Table: # | file:line | leakage type | why | severity | fix.
## Confirmation tests
For each high-severity finding: the test and, if run, the result.
## Clean areas
Bullets of what was checked and found clean.
## Open questions
Numbered, or "None".
</output_format>可替换内容
- {{code}}
- {{context}}
- {{> guardrails/investigate-before-answering}}
分类与标签
编程开发、机器学习、数据泄漏、训练评估、代码审查