Mirror, Mirror, Who’s the Best Forecaster of Them All?

Say you need an accurate forecast of future GDP or inflation. What’s your best bet—the economist who was hot last year or the forecaster in the middle? The record indicates it’s tough to consistently beat the median prediction.

Flip through the pages of any business periodical and you are likely to find seemingly precise economic predictions from various experts. They tell us that GDP will grow solidly—or not—and how much inflation we should expect. In this Commentary, we ask a simple question: Are there any demonstrably superior forecasters out there? To answer that question, we examine economists’ year-ahead growth and inflation predictions since 1983 to see whether any have distinguished themselves as particularly good (or bad) forecasters over time.

We find little evidence that any forecaster consistently predicts better than the consensus (median) forecast and, further, we find that forecasters who gave better-than-average predictions in one year were unable to sustain their superior forecasting performance—at least no more than random chance would suggest.

In the Eye of the Beholder

What does it mean to be a “good” economic forecaster? Well, it means to be accurate, of course. But what constitutes accuracy? In table 1, we summarize the track record of the median economist’s year-ahead predictions for real GDP growth and CPI inflation since 1983. (Forecasts were compiled by the Livingston Survey.) If we arbitrarily define an accurate prediction as being within 1/2 percentage point of the realized outcome, we would say that since 1983 the median forecast was accurate in only seven years, or about 30 percent of the time. In eleven of these years—a little less than half of the time—the median economist missed the mark in their growth forecasts by more than a percentage point. And 22 percent of the time, economists’ growth forecasts erred by more than 2 percentage points. The accuracy of the median forecaster’s prediction of inflation was a bit better over the 23-year period. Inflation predictions were accurate—that is, within 1/2 percentage point of actual inflation—39 percent of the time (nine of the 23 years). However, economists were off in their inflation predictions by more than a percentage point about 30 percent of the time.

Our definition of accurate—within 1/2 percentage point of the actual outcome—was arbitrary. Isn’t there a more objective standard we can use to judge forecast accuracy? Well, accuracy is always in the eye of the beholder, but one measure of forecast accuracy that is popular among statisticians is the “root mean squared error” (RMSE) of the forecast. A RMSE allows us to compute a confidence interval around a forecast based on its record of accuracy.

Applying this standard to our group of forecasts, we see that the RMSE of the median economist’s GDP forecast was 1.4 percentage points. Combining this information and a bit of statistical theory, we would expect the year-ahead GDP prediction of the median forecast to be within about 2.4 percent of the actual GDP growth rate about 90 percent of the time. So suppose the median forecaster expects the economy to grow 3.4 percent next year (its average since 1983). You could conclude—with 90 percent confidence—that the economy will grow between a robust 5.8 percent and a sluggish 1 percent. Similarly, the RMSE of the median economist’s inflation prediction over this period was 1 percent, which means that given an average inflation rate of 3.1 percent, you could be about 90 percent confident that prices will rise between a stable 1.4 percent and an uncomfortably rapid 4.8 percent over the coming year. The median forecast, in fact, encompasses a broad range of very different economic environments, and if you need to know next year’s economic environment with more precision than this, the recent track record of economic forecasters suggests you may be out of luck.

And the Winner Is…

The evidence presented above makes one thing very clear—we must be cautious before relying on what the median economic forecaster predicts, at least with respect to overall GDP growth and CPI inflation. But aren’t there some economists who tend to make pretty accurate predictions about the economy?

We examine this question by considering the accuracy of forecasters who provided at least six year-ahead predictions—the minimum number from which a statistically reliable statement regarding each forecaster’s accuracy could be made. This yields a sample of 79 real GDP growth forecasters and 74 inflation forecasters. We first test how many forecasters were able to produce a forecast that was superior to a “naïve” prediction. A naïve forecast is based on no understanding of the factors influencing the economy, and merely predicts that next year’s outcome will be the same as this year’s outcome. Such a forecast is analogous to a coin flip in the sense that the prediction assumes that next year’s outcome is just as likely to be higher as it is lower than this year’s outcome.

How did the economists fare relative to the naïve prediction? We found that a little less than half of the forecasters beat the naïve prediction with respect to growth (by which we mean they produced a lower RMSE). That is, a little more than half (53 percent) made worse predictions, on average, than the naïve forecast. With respect to inflation, 59 percent of the forecasters beat the random prediction (on average), while 41 percent did not.

We next apply a more difficult standard: Do any economists consistently beat the median forecaster? In other words, what proportion of economists was able to beat the median economist’s prediction over time? Figures 1 and 2 show the RMSE for each forecaster in our sample relative to the RMSE of the median forecaster. Of the 79 real GDP forecasters, only 22 (28 percent) in our sample had a better forecasting record than the median prediction, while a mere 14 of the 74 CPI inflation forecasters (19 percent) were superior to the median. Furthermore, of that select group who seemed to predict better than the median, their forecasting superiority was relatively small (less than 0.4 percentage point, on average)—a statistically insignificant advantage.

Experimenting further, we recorded each individual forecast’s percentile—the proportion of forecasts that were worse than the individual prediction. For example, if there were 51 forecasts for real GDP in some particular year, we assigned a score of 100 to the best forecast, a score of 98 to the second best forecast, and downward by 2 percentage points until the worst forecast earned a 0. The median forecast error would earn a 50 on this accuracy scale. We computed the average percentile rank for every economist who gave at least six annual growth and inflation forecasts. A scatter plot of this experiment is shown in figure 3.

Note that the points on the scatter plot are roughly aligned along the northeast to southwest diagonal, which means that economists who tend to be relatively better (or worse) at predicting GDP growth also tend to make relatively more (or less) accurate inflation predictions. (This observation is borne out statistically as a correlation coefficient of 0.4 percent, which is significant at the 99 percent confidence level for our sample size.) But central to the question posed in this Commentary, consider the distribution of points. About one-third of forecasters are represented by points in the top right quadrant of the graph (relatively adept growth and inflation forecasters), while roughly one-quarter of forecasters are represented by points in the bottom left quadrant (relatively poor growth and inflation predictors.) But only two of the best of these forecasters (those 21 in the northeast quadrant) could be distinguished statistically as being superior to the other forecasters, about the number one would expect given random chance. However, we did find six unusually bad forecasters, more than random chance would have suggested.

Let Me See You Do That Again…

As a final experiment, we wanted to see if forecasters ever got “hot” (or cold). Did forecasters who had a relatively good prediction in one year repeat that success with their next prediction? Our experiment was simple: We compared each prediction in our sample with the realized outcome and recorded whether the prediction was in the top half or bottom half in terms of accuracy. What percentage of forecasters who gave a better-than-average prediction in one year (were in the upper half in terms of accuracy) repeated that performance in the following year? If being a good forecaster were purely a random event, each year we would expect the chance of being in the upper tail of the accuracy distribution to be around 50 percent (an odd number of forecasters in some cases causes the deviation from exactly 50 percent.)

In table 2, we show that this appears to be the case. Of those forecasters who gave a better-than-median real GDP prediction in any particular year, only 46.8 percent were able to remain in the top half of the forecasters in the next year—just a bit under the 49.3 percent we would expect if forecast accuracy was purely random among the forecasters. The results regarding inflation predictions were similar to those for economic growth, as were those for the likelihood of repeat performances from forecasters who gave worse-than-median predictions.

In short, our results failed to reveal much persistence in relative forecast ability for year-ahead predictions of real GDP growth or inflation. In fact, what we find most clearly is that the relative accuracy of an economic prediction in one year does not make it any more likely that a prediction will be relatively accurate in successive years than what random chance would have indicated.

Are They Even Trying?

In virtually every way, the work we present here has been confirmed by many before us. In the 1980s, Stephen McNees documented the superiority of consensus forecasts relative to most individual forecasts, a result that has been upheld time and again. Further, many others have failed to find evidence that some forecasters possess a special insight into the economy that allows them to predict its course better than others.

One major assumption of these analyses is that an economist’s primary goal is, in fact, to make an accurate forecast. However, it may be the case that a forecaster has an objective other than accuracy. For example, Owen Lamont has shown that once a forecaster becomes more established, his or her predictions become more extreme and less accurate. Similar evidence suggests that some forecasters appear to seek publicity by being outliers, and that this publicity is of greater value than the relative accuracy of their predictions (David Laster, Paul Bennett, and In Sun Geoum). These ideas have been reworked more recently by Jordi Pons-Novell, who shows that forecasters from different institutional backgrounds appear to be motivated by different incentives, such that investment bankers may be trying to maximize publicity or other similar criteria, while forecasters who work at nonfinancial corporations seem to mimic more closely the consensus prediction.

Regardless of what motivates the forecaster, our investigation suggests that over the past 23 years, economists have had trouble producing forecasts that were superior to naïve predictions, and only a small proportion of forecasters were more accurate than the median forecast—and none statistically so. Finally, while some economists may distinguish themselves as being more accurate than their colleagues, none of the economists in our sample was able to demonstrate consistent superiority in forecasting accuracy.
So should you rely on economic forecasts for a description of what the future may bring? Well, we suggest you do so with caution. Our analysis suggests that the median economist’s prediction is as good a bet about the future as any other opinion, but remember, the confidence interval around that median forecast includes a very wide range of outcomes.

Recommended Reading

For work similar to that presented here, see:

• Stephen K. McNees. 1987. “Consensus Forecasts: Tyranny of the Majority?” Federal Reserve Bank of Boston, New England Economic Review (November/December).
• David Laster, Paul Bennett, and In Sun Geoum. 1999. “Rational Bias in Macroeconomic Forecasts,” Quarterly Journal of Economics (February).
• Owen Lamont, 1995. “Macroeconomic Forecasts and Microeconomic Forecasters,” National Bureau of Economic Research Working Paper no. 5284.
• Jordi Pons-Novell. 2003. “Strategic Bias, Herding Behavior and Economic Forecasts,” Journal of Forecasting (vol. 22).

Suggested citation: Bryan, Michael F., and Linsey Molloy, 2007. "Mirror, Mirror, Who’s the Best Forecaster of Them All?," Federal Reserve Bank of Cleveland, Economic Commentary no. 20070315