A probability is only useful if it means something. When a forecaster says 70%, the things they call 70% should happen about 70% of the time. Scoring rules turn that idea into a number you can track. Prolepsia uses two of the most common ones: the Brier score and log loss. For both, lower is better.
The Brier score
For a yes-or-no question, the Brier score is the squared distance between the probability and what happened, counting "yes" as 1 and "no" as 0.
| Forecast of yes | What happened | Brier score |
|---|---|---|
| 70% | Yes | (0.70 − 1)² = 0.09 |
| 70% | No | (0.70 − 0)² = 0.49 |
| 50% | Either | 0.25 |
| 95% | No | 0.9025 |
A forecaster who always says 50% scores 0.25 on every question. That is a useful reference point: a score below 0.25 on yes-or-no questions means the forecasts carried more information than a coin flip. A perfect forecaster, certain and always right, scores 0.
For a multiple-choice question, the Brier score adds the squared distance for every option, with 1 for the option that happened and 0 for the others. It runs from 0 to 2.
Log loss
Log loss scores a forecast by the probability it gave to what actually happened: the score is the negative logarithm of that probability. Giving 90% to the outcome that happened costs about 0.105. Giving 10% to it costs about 2.303. Giving it close to 0% costs a great deal.
Log loss punishes a confident miss much harder than the Brier score does. That makes it a good check on overconfidence. A forecaster who says 99% must be right almost every time.
Numbers and dates
For a question whose answer is a number or a date, a forecast is a range of values with a probability on each. These are scored with rules that charge more the farther the outcome lands from where the forecast put its probability. A forecast that spreads its probability widely pays a small cost even when it is right, and a narrow forecast that misses pays a large one.
What cannot be scored
A forecast can be scored once its outcome is established. Some questions never get there: no outcome is established, or the question is annulled because its terms stopped making sense. A score is only meaningful next to a statement of what it covers. Any record should say how many questions it includes, which it leaves out and why, and which forecast it scores on each question.
Which forecast counts
A question can have many forecasts over its life. Scoring only the last one flatters the record, because the last forecast is often made close to the outcome, when the answer is easier to see. Averaging the score of every forecast made before the question closed gives a fairer picture of how useful the forecasts were while the question was still open.
Prolepsia's own record
The track record lists every resolved public forecast with the outcome and its score, and states what it covers and what it leaves out. The methodology explains how each forecast is made.