Accuracy and Skill in Predicting Sporting Outcomes

[UPDATE 3/16: Dan Johnson replies to this post in the comments here.]

With March Madness about to tip-off and people paying attention to their bracket predictions, it seems like a good time to revisit the issue of predictive accuracy using sporting events as a laboratory for understanding.

Earlier this week I saw an article in the WSJ about some interesting research done by Daniel Johnson, an economist at Colorado College, who has developed a methodology to predict the winning of Olympic medals by nation. There is plenty of details here including data and a link to a peer reviewed paper (Johnson and Ali 2004).

Here is an excerpt from that WSJ article:

In what has become a biennial pre-Olympic event, Daniel Johnson, an economist at Colorado College, released his projections Monday with his usual caveat—his projections, which averaged a 93% success rate for overall medals from 2000-2010, pay exactly zero attention to the actual athletes who are participating in the upcoming Summer Games.

I contacted Johnson, and he was really great in answering questions and providing data (Thanks, Dan!). I was intrigued by the claims of predictive success of his methodology, such as this from Canada's National Post: "For a mathematical model compiled from freely available data, its predictive power is striking."

The metric used by Johnson to characterize predictive accuracy is a correlation between the prediction and the outcome. As I have written about on many occasions, predictions are best measured by skill, not accuracy and certainly not by correlation. Consider the useful anecdote that if you were to predict no tornadoes for, say, central Oklahoma every day, you'd have about a 99% accuracy rate and an eye-popping correlation. But you'd also provide no value added whatsoever.

To provide value added in forecasting any methodology must outperform some naive baseline expectation -- that is, a simple prediction. For Olympic medals there are a number of different metrics that one might use as a naive baseline. In this exercise using Johnson's predictions, I have used medal results from the prior Olympic Games as the basis for a naive forecast of results for the subsequent games. That is, a simple expectation for the 2004 Olympic Games medal counts would just be the results from the 2000 games, for the 2008 games the naive expectation would be the results from 2004 and so on. Any methodology purporting to have predictive skill ought to be able to improve upon this very simple naive baseline.

To evaluate a prediction against the naive baseline one can compute what is called a root mean squared error, which is simply calculated by taking the difference between the prediction (and the naive baseline) and the actual results for each country, adding them up, and then taking the square root to return to the original units.

For instance, the US had 97 total medals in the 2000 Sydney Olympics. Johnson's methodology predicted 103 for the 2004 Athens games. The actual number of medals won was 103. So the RMSE for the naive forecast was 6 and for Johnson was 0, meaning that in that instance, Johnson's methodology outperformed the naive baseline.

I have repeated this calculation for the top 20 medal-winning countries for 2004 and 2008, and compared the results of the naive forecast to the contemporaneous predictions made using Johnson's methods. Here are those results in the following graph:

What this graphs shows is, unfortunately, Johnson's predictions were not able to outperform a very simple naive baseline. The black bars are smaller than the red bars for both 2004 and 2008, indicating that the predictive error of Johnson's forecasts was larger than that of the naive baseline expectation. In other words, if you went to Las Vegas to bet on Olympic outcomes, you'd fare better using the naive baseline as a predictor than the methodology proposed by Johnson.

While I haven't comprehensively evaluated Johnson's predictions (he also predicts participation and Gold medals for all participating countries), my tentative conclusion is that Johnson's work may tell us something about relationships between different characteristics of nations and Olympic outcomes, but it does not appear to provide us with a skillful way to forecast Olympic medals. To make that case, he would have to utilize conventional metrics of forecast verification to demonstrate skill versus a naive baseline. As they say, prediction is hard, especially about the future!