The Least Thing in Indianomix

Vivek Dehejia, of Carleton University in Ottawa, and Rupa Subramanya, of the WSJ, have a new book coming out this week titled Indianomix: Making Sense of Modern India” (Random House). In it they cite an analysis performed here on The Least Thing. Here is an excerpt from an excerpt, published in the WSJ:

From astrology to tarot, human history is littered with our attempts at better predicting the future. More recently, these include more scientific (or at least apparently so) methods like polls and expert predictions. Philip Tetlock, a psychology professor at the University of Pennsylvania, is most famous for a 20-year study in which he tracked the predictions of hundreds of experts in many different fields spanning the range from business to international relations and then compared the actual outcomes to the predictions. He published his results in a landmark 2005 book called Expert Political Judgement: How Good Is It? How Can We Know. In the study, he found that the predictions of experts were no more accurate than if you’d just tossed a coin instead. In fact, sometimes tossing a coin would give you a better prediction than listening to an expert. As Tetlock colourfully puts it, even a monkey throwing darts at a dartboard would do better than most experts. You could probably do even better than the monkey, but that would be just by mechanically extrapolating a trend from today into the future. (The simplest form of extrapolation is what’s known in mathematics as ‘linear extrapolation’. Basically, if you have two points, you put a straight line through those points and make a prediction assuming that the next data point will also lie along that same straight line.)

There are innumerable examples of what Tetlock has in mind. Here’s one from March 2012. A much publicized paper by economics professor Dan Johnson of Colorado College and Ayfer Ali created a complicated statistical model with many variables to predict how countries would do in the medal tally of the winter and summer Olympic Games. They have a very large data set spanning games from 1952 to the present. The authors find some interesting correlations between variables, such as the intuitive ideas that rich countries win more medals than poor countries and that countries with snowy winters do better in the winter games. Where they falter is when they claim that their statistical model can provide ‘surprisingly accurate predictions’ of medal tallies in future games. But as Roger Pielke Jr, a professor of environmental studies at the University of Colorado, Boulder, showed, the interesting correlations that Johnson and Ali found didn’t translate into good predictive power. In line with Tetlock’s research, Pielke showed that the naïve assumption that a country would win as many medals next time as last time performs better than the complicated statistical model.

The analysis that they refer to was published here last summer. Further background can be found here. If experts cannot offer skill in predicting sporting outcomes -- closed systems with simple rules, and small number of actors -- then the prospects for offering skillful predictions in more complicated settings are slim indeed.

Fortunately, we'll always have Paul.