返回提示词目录第 1394 页

概率预测评分与校准复盘

依据已揭晓的预测记录计算评分与校准表,对照基准识别判断偏差,并给出受样本限制的改进方向。

完整提示词

<context>
A single forecast cannot be judged by its outcome: a 70% call that does not happen may still have been a good forecast. Forecasting skill shows only across many resolved questions. The Brier score (mean squared difference between probability and outcome, 0 is perfect, 0.25 is what always saying 50% earns on binary questions) rewards both calibration (things you call 70% happen about 70% of the time) and resolution (you separate likely from unlikely events instead of hugging the base rate). The log score punishes confident misses much harder. A calibration table needs enough questions per probability bin to mean anything; with a few dozen questions only large miscalibration is detectable. Scores are only comparable across forecasters on the same questions, and easy question sets flatter everyone, so compare with a benchmark: always 50%, or the base rate of the question set.

{{#forecasters}}Forecasters: {{forecasters}}{{/forecasters}}
</context>

<task>
<log>
{{forecast_log}}
</log>

1. Log checks: count resolved and unresolved questions, flag questions whose outcome or resolution criteria are ambiguous, and flag non-binary questions that need a different score. Exclude unresolved ones from scoring.
2. Scores: compute the Brier score overall and per forecaster, the log score if no probability is exactly 0 or 1 (say how you handle those), and the benchmarks (always 50% and the base rate of the set). Show the calculation for two rows.
3. Calibration table: bin forecasts (for example 0-10%, 10-30%, 30-50%, 50-70%, 70-90%, 90-100%), give the count, mean forecast and observed frequency per bin, and say which bins have too few questions to read.
4. Decompose if the set is large enough (roughly 50 or more): reliability, resolution and uncertainty, and say what each means here.
5. Patterns: overconfidence or underconfidence, extreme probabilities that missed, topics or time horizons where the forecaster does better or worse, and whether forecasts were updated.
6. How to improve: two to four concrete habits tied to the patterns found (start from base rates, avoid 0% and 100%, widen or narrow, update more often), and how many more questions are needed for firmer conclusions.
</task>

<constraints>
- Compute only from the log given; if outcomes are missing, do not guess them.
- Never call a forecaster good or bad from fewer than about 20 resolved questions; say what the data can and cannot show.
- Compare forecasters only on questions they all answered.
- Do not judge any single forecast as wrong because the less likely outcome happened.
{{> output/uncertainty}}
</constraints>

<output_format>
## Summary
Three sentences: overall score versus benchmarks, the main calibration pattern, and how much the sample supports it.

## Log checks
Bullets.

## Scores
Table: forecaster | questions | Brier | log score | benchmark Brier.

## Calibration table
Table: bin | count | mean forecast | observed frequency | readable?

## Patterns
Bullets.

## How to improve
Numbered habits.
</output_format>

可替换内容

  • {{#forecasters}}
  • {{forecasters}}
  • {{/forecasters}}
  • {{forecast_log}}
  • {{> output/uncertainty}}

分类与标签

思维提升、概率预测、校准评估、判断复盘、认知偏差

前往 AI 图片创作