Black Swans and Baseball

Two nights ago saw what many are calling the greatest final day in the baseball season ever, seeing off the end-of-season collapses of the Boston Red Sox and Atlanta Braves. The Economist (of all places) has a nice capsule summary of what transpired, and also provides an estimate of the near-impossibility of what transpired:

Let’s rehash. There was something on the order of an 0.5% chance the Red Sox would blow their nine-game lead over the Rays, and a 2% chance the Braves would lose their seven-and-a-half-game edge over the Cardinals. Then in tonight’s matchups, there was a 13% chance the Phillies would come back to beat Atlanta, a 5% chance the Orioles would come back to beat Boston, and an 0.3% chance the Rays would come back to beat New York. Multiply it all out, and the odds of witnessing what we just witnessed were worse than one in 500m.

Writing at the FT,  Jason Abbruzzese compares such numbers to the numbers that helped to create the global financial crisis:

The statistical revolution of the last 50 years has seriously impacted both the financial markets and professional sports. The Red Sox are among the teams that have attempted to most directly embrace this numbers revolution – often called Sabermetrics, which was written about by Michael Lewis in ‘Moneyball’ – in theatres now.

In finance, computer-based trading allowed the best and brightest to set their minds to figuring out just which algorithm can take advantage of increasingly complex financial instruments.

Looking back on each of these situations, we see that these numbers are often only as good as the people they’re built on and the people who built them. Statistics didn’t predict the fatal flaws in CDOs. Indeed, it was a formula that felled Wall Street.

Statistics did not predict the history-making crash of the Boston Red Sox – widely regarded before the season by statisticians and overweight sportswriters alike as the best team in baseball.

Nor could they. Statistics by their nature are backward-looking and can only give us an idea of what to expect.

But even though we know this to be the case, in the aftermath of witnessing an event that not long ago we would have dismissed as sheer fantasy, we reinterpret experience in a way that transforms the unexpected into a narrative of cause-and-effect that now makes any other outcome seem improbable from the vantage point offered by hindsight.  Again, Abbruzzese:

Down to their final out, Rays batter Dan Johnson, batting an anaemic .119 (that’s bad, cricket fans), with two strikes, launched a ball out of the park that tied a game that the Rays would go on to win.

The chances of that are small, to say the least, yet not a single person in Red Sox nation looks back at it with surprise. And that’s exactly how many of us look back at the financial crisis – it was inconceivable, but now none of us are surprised.

This dynamic is of course illustrates what Nassim Nicholas Taleb calls a black swan:

What we call here a Black Swan (and capitalize it) is an event with the following three attributes. First, it is an outlier, as it lies outside the realm of regular expectations, because nothing in the past can convincingly point to its possibility. Second, it carries an extreme impact. Third, in spite of its outlier status, human nature makes us concoct explanations for its occurrence after the fact, making it explainable and predictable.

I stop and summarize the triplet: rarity, extreme impact, and retrospective (though not prospective) predictability. A small number of Black Swans explains almost everything in our world, from the success of ideas and religions, to the dynamics of historical events, to elements of our own personal lives.

Our misperceptions of the role of chance in advance of and in the aftermath of rare events helps to explain why we some often make poor decisions, in and out of sport.