On 6 November we asked our forecaster ten questions whose answers were already public. What would five companies report for the quarter that ended on 30 September: earnings per share, and revenue?
All five had already reported. The test was to make the forecaster forget that. We gave each forecast an information date, from 8 July to 27 October, and held what it could see to that date. Then we compared its answers with what the companies reported.
This is a backtest. It cannot prove that anyone saw the future. It can show whether a method works when the future is held back properly. That part is harder than it sounds, so we start there.
How we made it forget
A forecaster can learn the answer in many quiet ways. Before this test we found and closed five of them in our own system. For each forecast:
- The date. The forecaster is told the information date as today's date.
- The data. Each data source answers as it would have on that date. SEC filings published later are not there, and news after the date is not there. Sources that cannot answer as of a past date are switched off.
- The tools. Tools that could reach today's figures refuse to run. That includes our analyst-estimate tools, so the forecaster could not look up consensus figures. In the 10 transcripts we read, it found none in the news either.
- Our own records. Forecasts we made after the date are hidden from it.
- The model. Every call went to GPT-6.1 Sol, whose training data ends on 30 April 2026, more than two months before our earliest date. That date comes from a published model card, not from OpenAI itself.
Then we checked. We read the full transcripts of 10 of the 60 forecasts by hand: every search, every document, every line of the forecaster's reasoning. None of the five earnings releases appears in them, and neither does any reported figure. Every dated web link in them is older than its forecast's information date. Ten of sixty is a sample, not proof. How a forecast with an information date works explains the method in more detail.
What we asked
Five companies, one from each fifth of the S&P 500 by size: Mosaic, Church & Dwight, AMETEK, Realty Income and GE Vernova. For each, two numbers as first printed in the earnings press release: earnings per share (GAAP, except Church & Dwight, where we asked for the adjusted figure the company leads with) and total revenue.
Each of the 10 questions was asked at 6 information dates: 8 July, 2 September, 30 September, and 14, 21 and 27 October. That makes 60 forecasts. Each is a full probability distribution, not a single number; below we use its middle value, the median. All 60 completed, in one day.
Against the obvious guess
The simplest forecast of a company's quarter is the same quarter last year. 49 of the 60 forecasts can be compared with it; the other 11 are explained below. The forecaster's median was closer to the reported figure than last year's figure in 45 of the 49. It called the direction of change, up or down from last year, correctly in 45 of 49. With information to 27 October, it was closer in all 8 that can be compared.
A slightly smarter mechanical rule, last year's figure scaled by the past year's growth, is harder to beat. The forecaster was closer than it in 30 of 44 comparable cases. Twelve of the 14 times the rule was closer were Realty Income, on both numbers at every date. That rule can also go badly wrong: for GE Vernova's earnings it said $13.81, against a reported $4.81.
Against Wall Street
We recorded the final analyst consensus for each number from news coverage on the day of each release. It is a hard comparison for us: those figures are the analysts' last word, set 1 to 7 days after our last information date.
Seven of the ten numbers are measured the same way by us and by the analysts. The other three are the earnings of Mosaic, AMETEK and Realty Income, where analysts estimate a different measure (adjusted earnings, or funds from operations for the property company) from the GAAP figure we asked about.
On two of the seven, AMETEK and GE Vernova revenue, the forecast gave no middle value to compare. The reason is under "What went wrong". The analysts were 3% to 5% low on both.
That leaves five. Our figures are the forecasts with information to 27 October. Where news services gave different consensus figures, the table shows the range:
| Number | Reported | Ours | Analysts | Closer |
|---|---|---|---|---|
| Church & Dwight adjusted EPS | $0.92 | $0.93 | $0.89 to $0.91 | Too close to call |
| Church & Dwight revenue | $1,602m | $1,597m | $1,579m to $1,594m | Us |
| Realty Income revenue | $1,622m | $1,599m | About $1,580m | Us |
| GE Vernova EPS | $4.81 | $3.63 | $4.00 to $4.03 | Analysts |
| Mosaic revenue | $3,101m | $2,967m | $3,080m | Analysts |
Two each, and one too close to call. Their two wins were wider than ours: across the five, our forecast missed by 6.4% on average and the analysts by 4.2% to 5.0%, depending on the news service. On five numbers we call that about level, and we would not read more into it than that. What makes it interesting is what each side spent to get there.
What it cost
A consensus figure is the average of estimates from several professional analysts; Zacks counted five for Mosaic. They follow each company through the quarter and can revise their estimates up to the day of the release.
Our forecaster worked alone, from public filings, data and news, with our analyst-estimate tools switched off and 1 to 7 days less information than the final consensus. Each forecast took a median of 21 minutes. What it cost us to run, in model and data charges (not a price anyone pays), was $2.29 to $4.86 a forecast, with a median of $3.15. The ten forecasts with information to 27 October cost $31.50 together. All 60 ran on one day, between 10:24 and 22:42 UTC on 6 November, a few at a time, for $194.88.
So for about $3 and 20 minutes a number, it came out about level with the analysts on the five numbers we could compare. It was ahead of last year's figure, and ahead of the growth rule more often than not. That is the result we find most encouraging, and the one we most want to test on more companies.
It got better as the quarter went on
The most useful thing a forecaster can do is update. Mosaic shows it best. A year earlier the company earned $1.29 a share. In July the forecaster put the quarter at about break-even, $0.01. In early September it moved the wrong way, to a profit of $0.25. By 30 September its median was a loss of $0.20, and by late October a loss of $0.18. The range stayed wide: at the last date, from a loss of $0.83 to a profit of $0.36. Mosaic reported a loss of $0.23. Its release pointed to lower sales volumes, partly because it cut phosphate production when affordable sulfur was scarce, and to higher raw material costs.
Across all five companies, the average earnings miss, as a share of each question's answer range, fell from 14% in July to 9 to 10% from the end of September on.
What went wrong
It ran low. In 48 forecasts the result fell inside the question's answer range. Of those, 39 landed inside the forecast's 80% range. That is close to the 80% we aim for. But every one of the 9 misses was on the same side: the company came in above the forecast. The 9 come from just two numbers: Realty Income's earnings did so at all six dates, and GE Vernova's at three. GE Vernova reported $4.81 a share. The last forecast said $3.63. The analysts said about $4.
Two answer ranges were too narrow. Each question has an answer range built around last year's figure. For AMETEK and GE Vernova revenue, the range topped out too low: AMETEK closed a large acquisition during the quarter and GE Vernova grew 27%. By late October the forecaster put 82% (AMETEK) and 94% (GE Vernova) of its probability above the top of the range, and both companies did come in above it. That was the right call, but a median above the top cannot be compared, so these 11 forecasts are left out of the counts above. (AMETEK's July forecast still had a median inside the range, so it is compared with last year's figure. All 12 forecasts for these two numbers are left out of the 80% range count, which is why it starts from 48.) That is our design error, not the forecaster's.
What this does not show
- Five companies is a small sample, drawn from an S&P 500 list about two years old. The count against last year's figure is clear. The draw with analysts, on five numbers, is encouraging but not proof that we match them in general.
- The 60 forecasts are ten numbers asked at six dates, so they are not 60 separate tests. The four times last year's figure was closer involve two numbers, and so do the nine misses above the forecast's range.
- A backtest can leak in ways we did not find. We read 10 of 60 transcripts in full.
- We ran these forecasts after the results were out. Only forecasts made before results exist can prove foresight. That is why our public forecasts are sealed with a published hash every month: see the monthly snapshots.
The data
Download all 60 forecasts (CSV): the 10th, 50th and 90th percentiles of each, last year's figure, the reported figure, and the question's range. A percentile left empty lies beyond the range; the last two columns give the probability below and above it. Reported figures are from each company's earnings press release on SEC EDGAR. Analyst consensus is from release-day coverage by Zacks (through AP and Yahoo Finance), LSEG, FactSet and others; where providers disagree, the table shows the range.
Where each analyst figure in the table comes from:
- Church & Dwight earnings: LSEG I/B/E/S and Chartmill.
- Church & Dwight revenue: GuruFocus and Chartmill.
- Realty Income revenue: Zacks on Yahoo Finance, worked back from the 2.6% beat it reports.
- GE Vernova earnings: Zacks and StreetInsider.
- Mosaic revenue: Zacks through AP and Zacks on Yahoo Finance.
A study with 50 companies would tell us much more. If we run it, we will publish it the same way, misses included.
This is research about our forecasting method, not investment advice.
Your Prolepsia Team