Testing IPCC Projections Against What Happened

Arguments for or against doing things to slow climate change depend on what will happen if we don’t, a question the IPCC reports try to answer. That makes it important to know how reliable their predictions are. The scientific section of the latest report (IPCC AR6 WGI) runs to almost four thousand pages, largely of detailed analysis, depending on multiple scholarly articles for each step — Chapter 7, to pick one at random, has fifty editors and about nine hundred articles in its list of references. Someone with unlimited time, energy and expertise might be able to go through all of the calculations that produced the predictions in the reports in order to see if they were done correctly, but that is not a practical option, which raises the problem I mentioned in an earlier post — how can the reader know whether to trust calculations he cannot work through for himself?

The climate system is too complicated to make predictions on the basis of theory alone possible, hence the IPCC project largely consists of sophisticated curve fitting, picking a form for the relationships among observables suggested by physical theory, choosing parameters for the relationships, how strong each effect is, by finding the values that best fit historical data.  With enough tweaking of the models and adjusting of parameters that process can fit past data but doing so does not tell you whether the models will correctly predict future data. With enough parameters you can fit the skyline of New York.

The solution is to test the model against data that not used in creating it. We do not know the future, the future eventually becomes the past, so a model constructed in 1990 can be tested in 2021 against data that did not exist when the model was constructed.

The past IPCC reports are webbed. Back in 2014, after observing people on one side of the argument claiming that the models did a good job of predicting temperature changes, people on the other side that they did a terrible job,  I looked at each of the first four reports to see what someone who read it would expect future temperature to do and reported the results on my blog. If you would like to check my conclusions about what each report implied for yourself you can find links to the reports here.

What the IPCC Predicted

The executive summary of the first report (1990) contains:

Under the IPCC Business-as-Usual (Scenario A) emissions of greenhouse gases, the average rate of increase of global mean temperature during the next century is estimated to be about 0.3°C per decade (with an uncertainty range of 0.2°C to 0.5°C).

The graph shown for the increase is close to a straight line at least from 2000 on, so it seems reasonable to ask whether the average increase from 1990 to the present is within that range. 

Figure 18 from the Second Assessment Report (1995) shows the future temperature through 2020. Through that date, it rises steadily at about .14°C/decade.¹

From the Third Assessment Report (2001):²

For the periods 1990 to 2025 and 1990 to 2050, the projected increases are 0.4 to 1.1°C and 0.8 to 2.6°C, respectively.

For the former period, that implies an increase of from .11 to .31 °C/decade.

The Fourth Assessment Report (2007) has:³

For the next two decades a warming of about 0.2°C per decade is projected for a range of SRES emissions scenarios.

What Happened

When I did the calculations in 2014 I found that the IPCC had predicted high four times out of four, twice by enough so that actual warming was below the bottom of the predicted range. That looked like evidence that we should not put much weight on their predictions of future temperature. As of 2014 the critics were right.

Currently, the last year whose temperature is shown on the NASA page I use for temperature data is 2018; I redid the calculations accordingly. Here are the results:

The first IPCC report was released in 1990. From then to 2018, global temperature rose .38°C for an average of .14°C/decade, well below the predicted range.

The second report was released in 1995. From then to 2018, temperature rose by .37°C, for an average rate of growth of .16 °C, a little higher than the prediction.

The third report was released in 2001. From then to 2018, temperature rose by .29°C for an average of .17°C/decade, towards the lower end of the predicted range.

The fourth report was released in 2007. From then until 2018, temperature rose by .18 degrees, .16°C/decade, a little below the predicted .2°C/decade.

The predictions look better now than they did in 2014, high three times out of four, low once, and only once, with the first report, has actual warming been below the predicted range. They are still running a little hot but the results look consistent with random error. That makes it at least possible that the IPCC researchers are now modeling the climate system well enough to produce reasonable estimates of its future behavior.

It is possible but far from certain because the test the predictions pass is not a very strong one. A theory that correctly predicted the outcome of next year’s elections, including the Presidency and every house and senate seat, would be a very good theory indeed, since doing that well by chance is very unlikely; we would have good reason to trust its future predictions. A theory which correctly predicted which party will end up with a senate majority after the 2022 election would be better than one that got it wrong but not much better, since one can get the right answer half the time by flipping a coin.

The IPCC reports rely on complicated models and a lot of data. One way to judge how good a job they have done of modelling climate is to compare their results with those of much simpler models. The simplest plausible model is a linear regression, fitting the data to a straight line. Looking at the graph, warming starts about 1910, stops about 1934, restarts about 1965. Linear regressions starting in 1910 give a very poor fit so instead I did them starting in 1965. In 1990 the simple model would be a linear fit of temperature from 1965 to 1990.⁴ The slope of that line is .18°C/decade, so that is what the simple model predicts for warming thereafter. I repeated the calculation for the dates of the other three reports.

The table shows the results. “Regression slope” is the slope of the regression line, hence the rate of warming it shows. “Projected slope” is the slope implied by the IPCC report,⁵ “actual slope the rate of warming from the date of the report to 2018. All warming rates are in °C per decade.

The warming rate predicted by the regression beats the IPCC prediction four times out of four. The results of the models no longer show a clear bias but they don’t tell us much more than that Earth is warming, which we already knew.

They are, however, doing better than they were in 2014.⁶

1. The figure is on page 323 of Climate Change 1995 The Science of Climate Change. When I did my calculations in 2014 I thought it was .13°/decade but measuring the graph more carefully I now think it is .14.

2. Page 60 of Climate Change 2001 Synthesis Report.

3. Page 7 of Climate Change 2007 Synthesis Report.

4. Strictly speaking I should have run the regression to whatever was the last year whose temperature was known in 1990 but since I didn’t know what that was I used 1990 instead and similarly for the other three reports.

5. Where the report gave a range I took the midpoint.

6. An article that adjusts the predictions of the models to take account of deviations of the actual input numbers, such as CO2 concentration, from the numbers they assumed, finds that the errors range from 8% to 28%. My regression result do better than that too.