Why can economic forecasts go wrong?

Economic forecasting seeks to predict how output, unemployment, productivity and other key indicators will develop over the coming months and years. While forecasts can be thrown off course by unexpected events like wars and pandemics, slowly evolving changes can also lead to forecast inaccuracy.

In November 2021, the Bank of England Monetary Policy Report forecast that the UK’s annual rate of inflation would be 3.4% at the end of 2022: in fact, it rose to 9.2%. Similarly, in December 2021, the European Central Bank forecast eurozone inflation of 3.2% for the end of 2022: just as in the UK, the realised rate was 9.2% in December 2022.

Was this recent serious mis-forecasting of inflation by central banks due to bad analysis or poor judgement? Or did central banks fail to account for feedbacks from wage increases (‘wage-price spirals’) or did they have excess faith in inflation expectations being grounded at 2%?

This article sets out some of the reasons why economic forecasts can end up being inaccurate – and what the implications are for the practice of forecasting and its uses for policy-making and planning.

Can short-term economic forecasts really be ‘badly wrong’?

Sadly, yes. The unanticipated and dramatic impact of the Covid-19 pandemic is a powerful illustration of how economic forecasts can be very wrong.

Imagine that it is December 2019, and you are trying to predict a variety of measures of the UK economy for 2020, such as monthly output (measured by gross domestic product, GDP) and unemployment.

Figure 1 shows a graph of 12-month ahead GDP forecasts from a simple but widely used model that relates the next forecast to the previous outcome. Experience has shown that this model usually does reasonably well in forecasting competitions, where large numbers of different methods are compared for their accuracy.

Figure 1: 12-month ahead forecasts of GDP from December 2019

Source: Office for National Statistics (ONS), 2021
Note: UK monthly GDP is a chained volume index, 2016=100, of gross value added (GVA) vintage updated to March 2021 in June 2021.Forecasts for 2020(1)-2020(12) from a first-order autoregression estimated by least squares over 2017(1)-2019(12).

Figure 1 also shows the interval forecasts based on ±2 'standard errors' (a metric used by economists and statisticians to measure how accurate the forecasts are compared with the true value). The forecast period lies to the right of the vertical line and at the time would have been in the future.

Here the interval aims to include the outcomes 95% of the time: their widening reflects that further ahead is more uncertain.

In 'normal times', such forecasts would be reasonably accurate. But they were dramatically far from what actually happened after the pandemic struck and lockdown measures were imposed (see Figure 2). The interval forecasts now look tiny beside the vast drop in GDP.

Figure 2: Forecasts of GDP from December 2019 compared with outcomes

Source: ONS, 2021
Note: UK monthly GDP is a chained volume index, 2016=100, of GVA vintage updated to March 2021 in June 2021. Forecasts for 2020(1)-2020(12) from a first-order autoregression estimated by least squares over 2017(1)-2019(12).

The forecasts also seriously underestimated the increase in unemployment as a result of Covid-19 (see Figure 3).

Figure 3: 12-month ahead forecasts of unemployment from December 2019

Source: Labour Force Survey: International Labour Organization (ILO) unemployment rate, UK all aged 16 and over (%); not seasonally adjusted: series MGUK
Note: The forecasts are from Castle et al, 2021

Again, these forecasts are based on the same simple form of forecasting model. In practice, more sophisticated models are used for forecasting both GDP and unemployment as these are related: falls in GDP usually lead to rises in unemployment (an issue to which we return below).

Do such bad forecasts only happen after crises?

The forecasts in Figure 2 are among the worst errors ever, but bad forecasts are all too common. Many unanticipated events can cause what we call 'forecast failure', when the outcomes are far from the forecasts measured in standard errors.

The basic cause is a 'location shift' in the data after a forecast has been made. A location shift is when the average value of the data unexpectedly changes. It is hardly surprising that large shifts occurring after a forecast has been announced lead to serious forecast errors. But slowly evolving changes can also be problematic, as we show next.

What else can cause bad forecasts?

A shift in a trend that departs gradually from previous growth rates can also cause systematic forecast failure. An example of this is what happened to productivity forecasts by the Office for Budget Responsibility (OBR) after the global financial crisis of 2007-09 (see Figure 4).

Here, the forecasts are successive five-year ahead predictions made semi-annually. They are measured as output per person per hours worked. Looking at Figure 4 now, there is an obvious shift in the trend around 2011, but that was unclear at the time.

Figure 4 also shows the `real-time’, or flash, data, which were those available to forecasters at the time, rather than the final measures produced after various data revisions and updates.

Flash output data can be poorly measured as not all firms’ results are reported by the time needed to construct the aggregate. Consequently, statistical rules are used to `in-fill’ these missing observations. After a shift has occurred, previous in-fill rules would not take that shift into account, leading to measurement errors.

The basic problem in Figure 4 is clear: the OBR continued to forecast using the pre-2008 trend rate of growth of productivity long after it had stopped. This assumption is a form of 'equilibrium correction' where the economy is assumed to return to its previous state or growth after a shock has passed but that did not happen. The slower growth remains unexplained, but has persisted.

The result is a 'hedgehog graph' where the forecast lines look like the spines on the back of a hedgehog. All the forecasts’ spines point in the same direction showing that the forecasters are predicting the past trend to continue, when in fact it does not.

Hedgehog graphs are common, which suggests that after shifts, forecasters stick to their forecasting models beyond their 'use-by date'. Other mistakes in forecasting models can worsen forecasts but rarely cause failure.

Figure 4: OBR forecasts of UK productivity

Source: Martinez et al, 2021
Note: UK productivity (output per hour worked). Historical releases from the ONS and the National Archives. Output is measured as GVA excluding oil and gas in millions of pounds (ONS code UIZY or KLS2) while hours worked are measured as total actual weekly hours worked (ONS code YBUS). Both variables are seasonally adjusted. All vintages are re-normalised to 2009 Q1=100. The forecasts are close approximations to the OBR’s five-year ahead forecasts of productivity made in the first and third quarters of each year.

What can be done to avoid systematic mis-forecasting?

So, how should we make economic forecasts, given all these challenges? Importantly, the systematic nature of bad forecasts as seen in Figure 4 can be corrected by modifying the way in which forecasts are made so they are more robust after shifts.

Theoretical analyses of methods of forecasting have revealed that shifts in the 'equilibrium mean' (the long-run value about which a variable moves, like a 5% unemployment rate) or changes to long-run trends are the main sources of systematic forecast failure. This failure occurs when models continue to forecast the pre-shift mean or trend, and so are systematically wrong.

Consequently, rapidly 'correcting' shifts in the equilibrium mean or trend can be very helpful. In one approach, forecasts are made after replacing the previous equilibrium mean with a temporary new mean estimated using only very recent data.

For example, despite the numbers of coronavirus cases and deaths changing very rapidly during 2020, accurate short-term forecasts were possible by always quickly updating the model.

Figure 5 shows one example of what is called a smooth-robust predictor, where forecasts are smoothed after the transform to a temporary equilibrium mean to put them 'back on track' despite the trend shift.

As shown, the hedgehog effect is reduced substantially and the forecasts (which could have been made at the time) are greatly improved. There are many measures of forecast accuracy relative to outcomes, and one used commonly is the square root of the average (or mean) of the squared forecast errors (abbreviated as RMSFE): the square root makes the units of the RMSFE the same as that of the data.

For Figure 4 and 5 forecasts, the RMSFEs over the forecast period are 3.63 and 1.21 respectively, so using the smooth-robust predictor results in a three-fold reduction.

Figure 5: Robust forecasts of UK productivity

Source: Martinez et al, 2021
Note: UK productivity (output per hour worked). Historical releases from the ONS and the National Archives. Output is measured as GVA excluding oil and gas in millions of pounds (ONS code UIZY or KLS2) while hours worked are measured as total actual weekly hours worked (ONS code YBUS). Both variables are seasonally adjusted. All vintages are re-normalised to 2009 Q1=100.

Are conventional economic forecasts useless?

Not at all. Certainly, little can be done when forecasting before unexpected large shifts, as seen for energy prices after the Russian invasion of Ukraine.

Even so, the apparently dreadful forecasts in Figure 1 have a use as counterfactuals to the impact of Covid-19. Had the pandemic not occurred, the forecasts would have been reasonably accurate, so the differences between them and the outcomes allow the impact of the pandemic to be measured.

This is really useful for unemployment forecasts that are based on an econometric model during the pandemic, as the benefits of the UK government furlough scheme can be calculated in the number of jobs saved.

Usually, a large fall in GDP leads to serious job losses as sales fall – and 2020 is easily the largest fall on record – but unemployment rose far less than might have been expected.

The furlough scheme policy intervention broke the usually constant econometric model relations by protecting people’s jobs and artificially holding down the measured unemployment rates, protecting lives and livelihoods in doing so.

Figure 6 shows successive three-month ahead forecasts from the end of March 2020 using an econometric model of unemployment and that model plus a measure for the furlough policy intervention.

The original model predicts a large jump in unemployment, but once the policy effect can be estimated, it comes back on track. It is perhaps even a little low, but far better than those in Figure 3 made before the pandemic struck.

Figure 6: Forecasts of UK unemployment from an econometric model with a furlough policy intervention

Source: UK unemployment rate (aged 16 and over, seasonally adjusted, from ONS Labour Market Survey, MGSX). The forecasts are from Castle et al, 2021.

Could bad forecasts merely be the outcomes of bad models?

Let’s define a bad model as one that differs substantively from the real-world process that actually operates. Although intuition would suggest that if you start with a bad model, your forecasts will reflect this and be poor, remarkably this is not always the case.

Bad models can indeed forecast poorly, but they can also forecast well. Similarly, good models can forecast well, but also poorly. This is not overly helpful – but both happen in practice, and they can be derived from carefully delineating in a taxonomy all possible sources of forecast errors.

This shows that only a few problems can lead to forecast failure, including equilibrium mean or trend shifts, badly measured forecast-origin values, and wrong forecasts of 'offline' (unmodelled) variables (such as oil prices, exchange rates, etc.).

These are all location shifts relative to the pre-existing model. So, incomplete models can forecast well until a relevant influence, not included in the model, changes. This means that we cannot judge the quality of a model by its forecast performance.

What explains the inaccuracy of recent UK inflation forecasts?

Inflation is affected by many factors, as Figure 7 shows (calculated in August 2022 and called a nowcast rather than a forecast because it predicts the present or very near future). The estimates are derived from solving a dynamic system of econometric equations for unemployment, wages, prices and productivity. Commodity price inflation, which includes energy prices, has the biggest impact.

Figure 7: UK inflation nowcasts for 2022

Source: Castle et al, 2022

The wage equation includes a wage-price spiral allowing for second-round effects (where wage increases feed back to raise prices), and the resulting forecast of 12.5% assumes 100% of price increases are passed on to wages, which further increases prices. In the event, only 80% was passed on.

As a result, our forecast is slightly above the actual outcome of 10%, but importantly does not suggest a temporary blip in inflation as initially assumed by central banks, and it emphasises wage-price interactions.

What else can cause forecast failure?

There are some myths to dispel. Economic forecasting can sometimes seem mystical, derived from complicated models or plucked out of thin air. A common myth is that not knowing the correct model and having to estimate the implied relationships based on past data can cause poor forecasts.

By itself, this is not the case. Having to estimate a model empirically from past data will add a bit of uncertainty to its forecasts but usually won’t cause forecast failure.

A second myth is that everything that is relevant must be included in the forecasting model. Rather, omitting hard-to-predict relevant offline variables can be better than including them.

A third myth is that simple models are hard to beat. But what is 'simple'? Forecasting by the past average of the data is a simple model, but it will produce badly biased forecasts when that mean shifts.

An alternative simple model (the one used above), which forecasts from the last data observation is often better, and the more complicated smooth robust approach is usually much better after shifts.

Simplicity is not a key characteristic of a good forecasting model, whereas robustness to changes in the long-run mean or trend of the data is. Forecasting principles can be formulated to guide the formulation of forecasting models based on experience with what methods do well in forecasting competitions and real-world forecasting practice.

Should we worry about poor forecasts?

The implications of bad forecasts can be immense: from poor decision-making to planning mistakes and ill-conceived policy. So, there is value in ensuring forecasts do not go badly wrong.

Yet even murky glasses are better than a blindfold. Our book, Forecasting: An Essential Introduction (written with Michael Clements), illustrates the myriad of potential problems that can confront forecasters facing a less than mundane road trip, highlighting those for which there are possible solutions, as well as the unpredictable events that crop up.

While forecasters should not beat themselves up about not foreseeing unanticipated events such as pandemics, wars and other crises, they should avoid systematic forecasting mistakes once such a shock has occurred. To predict such events needs soothsayers in possession of a crystal ball, whereas bad forecasts are those that are still poor after shifts have been observed.