The fastest way to test a forecasting method is to run it on questions that have already been settled. You ask it what it would have said on a past date, then compare its answer with what happened. This only works if the run cannot see anything published after that date. A forecast that read tomorrow's newspaper is not a forecast.

The leak is easy to miss

Information from after the date can reach a forecast in quiet ways:

  • A web search returns today's pages, including articles written after the outcome was known.
  • A statistics page shows today's figures, which include revisions made long after the date.
  • A spreadsheet has rows dated before the cutoff that were only published weeks later, because data is often published some time after the period it measures.
  • A page that looks old carries a "last updated" note or a link to a later event.

Each of these makes a past forecast look better than it really was, and the method looks more reliable than it is.

What Prolepsia does in a historical run

For a historical run, Prolepsia restricts its tools to sources that support a cutoff date, so the run does not read what was published after it.

  • No open web search. General web search and page reading have no reliable date limit, so a run with a cutoff date does not use them.
  • Dated data sources only. The data sources a historical run can use are the ones that can answer as of a date. Each applies the cutoff to its own data.
  • Your files are cut at the date. Rows in an uploaded data file dated on or after the cutoff day are dropped before Prolepsia reads the file. The run is also warned that a value may be published some time after the date it measures, so the latest rows before the cutoff may not have been public yet.

The limits

How well this works depends on the source. Not every source keeps its figures as they stood on the day, and uploaded data can reach you later than the dates in its rows, so its publication date matters. A historical run is a strong test, but not a perfect one. Check each source's limits before relying on a historical run, and treat a result that looks too good as a reason to look for a leak.

Why it matters for you

If you want to know how far to trust Prolepsia on a kind of question, a historical run on past questions of that kind gives you evidence before you rely on it for a live decision. The methodology describes how a forecast is made, and the track record shows how Prolepsia's live public forecasts have scored so far.