时间序列预测模型回测比较
按实际决策周期设计滚动回测,比较预测误差、偏差与稳定性,提供模型选择依据及代码。
完整提示词
<context>
Picking a forecast model by in-sample fit, or by a single train/test split, routinely picks the wrong one: complex models fit history better and forecast worse, and one test window can be a lucky or unlucky stretch. Practitioners use rolling-origin evaluation (time-series cross-validation): many forecast origins, each forecasting the real decision horizon with only data available at that origin, mirroring how the forecast will be used. They compare against a seasonal naive benchmark (MASE scales errors by it), look at bias as well as error because a consistently high forecast costs differently from a noisy one, check that 80% intervals actually cover about 80%, and look at how much each model's error swings between windows. When the difference between models is within the window-to-window noise, the simpler model or a simple average of the top two (forecast combination often beats any single model) is the safer pick.
</context>
<task>
Compare forecast models for this decision: {{decision}}
<series>
{{series}}
</series>
{{#candidates}}
<candidates>
{{candidates}}
</candidates>
{{/candidates}}
1. Design the backtest to match the decision: horizon, refresh cadence (step between origins), the minimum training window, and the number of origins (aim for at least 5 where history allows). State what information is available at each origin; drivers or regressors must use values known at that time.
2. Candidates: always include naive and seasonal naive; add the user's candidates, an ETS model and, where history is long enough, ARIMA. Add a simple combination (mean of the best two).
3. Metrics per model and per window: MAE (or WAPE across series), MASE relative to seasonal naive, bias (mean error and as a percent of actuals), and 80% interval coverage where the model gives intervals. Weight errors at the horizons that matter for the decision.
4. Summarise across windows: mean, spread (standard deviation or range) and the worst window for each model.
5. Choose: the model with the best error and acceptable bias and coverage; if two are within the spread, prefer the simpler or the combination. Say what result would change the choice.
6. If results tables are given instead of data, compute the summary from them and show the arithmetic.
7. Write {{tool}} code that runs the backtest and produces the results table (Python: statsforecast or sktime cross-validation; R: fable with stretch_tsibble).
</task>
<constraints>
- Never use data after an origin to fit, tune or build features for that origin.
- Do not select or tune on the same windows used to report the final comparison without saying so; if tuning is needed, use earlier windows for tuning.
- Do not declare a winner on a difference smaller than the window-to-window spread.
- Do not invent results; when data is not provided, give the code and the table structure.
{{> output/uncertainty}}
</constraints>
<output_format>
## Summary
The chosen model, by how much it beat the benchmark, and how confident the choice is.
## Backtest design
Origins, horizon, step, training window, information available at each origin.
## Results
Table: model | mean MAE or WAPE | MASE | bias | 80% coverage | spread across windows | worst window.
## Choice
The pick and the reasoning, plus what would change it.
## Code
One {{tool}} code block.
## Caveats
Bullets: few windows, structural breaks, events in some windows only.
</output_format>可替换内容
- {{decision}}
- {{series}}
- {{#candidates}}
- {{candidates}}
- {{/candidates}}
- {{tool}}
- {{> output/uncertainty}}
分类与标签
办公效率、时间序列、滚动回测、预测评估、模型比较