The Prediction Test
The two most famous theories in the field of forecasting are the butterfly effect, and the efficient market hypothesis. Both are theories, not of prediction, but of non-prediction.
The butterfly effect was developed by MIT meteorologist and chaos theorist Ed Lorenz in the 1960s. He found that computer simulations of a toy weather model tended to stray apart over time if the starting point was changed by even a tiny amount (chaos!). He proposed that this “sensitivity to initial condition” was a property, not just of his three-equation model, but of the weather itself. When he submitted an untitled talk for the 1972 conference of the American Association for the Advancement of Science, the person hosting the session supplied a provocative title: “Predictability; Does the Flap of a Butterfly’s Wings in Brazil Set Off a Tornado in Texas?”
The efficient market hypothesis was first proposed in a 1970 PhD thesis by Eugene Fama, from the University of Chicago. It says that price changes in financial markets are caused by random perturbations (e.g. news) and therefore follow a “random walk” which is inherently unpredictable.
Apart from fame, the theories have many other things in common. They both provide a scientific reason for forecast errors, such as the financial crisis. They both assume that forecast error is due to random effects (insects or news). Both theories – or at least their typical applications – assume that the underlying model of the system is correct. And they are both used to justify complicated techniques that are hard to interpret or falsify.
In the 1990s weather forecasters seized on the butterfly effect as an excuse for forecast error, but also as a rationale for elaborate “ensemble forecasting” schemes. Instead of making a single “point” forecast, an ensemble of forecasts is here generated from a set of perturbed initial conditions, and used to produce a statistical forecast that takes into account the effects of chaos. When forecasters made typical perturbations of the sort that might be produced by observational error, they found that the simulations didn’t diverge as quickly as expected, which was possibly a hint; however they soon found ways to select specially optimised perturbations which did exhibit the desired divergent behaviour.
The efficient market hypothesis meanwhile might have shown that price changes were unpredictable, but also enabled the use of statistical models which claimed to predict the probability of a price change, such as the Value at Risk model. In either case of course the statistical forecast is only valid if the underlying model of the system is correct.
Both theories are hard to disprove, and remarkably resilient to criticism. When I (David) showed in a 1999 presentation at the European Centre for Medium-Range Weather Forecasts that plots of forecast error show a square-root shape, which is characteristic not of chaos but of model error, I was contradicted by a number of people in the audience. The next day I received an email from one of the top research heads, which said that he had checked a plot of forecast errors, and, in stark contrast to my talk, “they certainly show positive curvature.” In other words, they were caused by chaos, not model error. We therefore decided that someone there should try to reproduce my results, by plotting the errors as a function of time.
When the results showed a near-perfect square-root shape, I received an email saying that “I guess it would be possible to get an initially square root shape from initial condition error if the error was initially in very very small scales which rapidly saturates but cascades up to produce errors of larger scale, which then saturate, but cascade up to produce errors of still larger scale.” (That was the exact point when my view of science began to shift.)
Similarly, as Andrew W. Lo and A. Craig MacKinlay wrote in their book A Non-Random Walk Down Wall Street: “One of the most common reactions to our early research was surprise and disbelief. Indeed, when we first presented our rejection of the Random Walk Hypothesis at an academic conference in 1986, our discussant – a distinguished economist and senior member of the profession – asserted with great confidence that we had made a programming error, for if our results were correct, this would imply tremendous profit opportunities in the stock market. Being too timid (and too junior) at the time, we responded weakly that our programming was quite solid thank you, and the ensuing debate quickly degenerated thereafter. Fortunately, others were able to replicate our findings exactly.”
Needless to say, both the butterfly effect and efficient market theory survived these and other challenges.
Finally, both theories rely on a kind of magical thinking – that the atmosphere is incredibly sensitive to the smallest change, so perturbations grow exponentially instead of just dissipating (try waving your hand in front of your face to see which is more physically realistic); or that the economy is magically self-correcting, like a door which snaps instantly shut after being opened.
One difference is that the butterfly effect does double duty in other areas such as economics. As then-Fed chairman Ben Bernanke explained in 2009, “a small cause – the flapping of a butterfly’s wings in Brazil – might conceivably have a disproportionately large effect – a typhoon in the Pacific” which was a useful thing to bring up after you just failed to predict the US housing crisis. However, the idea that unpredictability is caused by efficiency has failed to catch on outside of economics. For example, no one thinks that snow storms that come out of nowhere are efficient.
So why are these theories both still around? The reason is simple. As the physicist Richard Feynman once said, “The test of science is its ability to predict.” The magic of science is the ability to make it look like you can predict.