Skill in Prediction: Olympic Medals Edition

Not long ago I offered a critique of a statistical methodology used to predict Olympic medal outcomes. In my critique, I explained that a naive prediction -- the medals won in the previous Olympic games -- outperformed a sophisticated statistical model, as shown below.

What the graph indicates is that the statistical model had no skill -- it provided no value-added to the naive forecast.

At the FT, Martin Stabe is tracking four such models as follows:

• Emily Williams, a PhD candidate at London Business School updated a model first developed by  Andrew Bernard of the Tuck School of Business at Dartmouth for the 2000 games in Sydney. It focuses on population, income per capita, total GDP, and host-nation advantage to explain Olympic performace.
• Daniel Johnson, an economist at Colorado College has used a similar model to publish predictions for Olympics for several years. A change to his approach for the 2012 adds a host-nation effect that pre-dates and post-dates the current games, so China and Brazil should benefit in addition to Great Britain), and eliminates an earlier focus on political structures and climate.
• José Ursúa and Kamakshya Trivedi of Goldman Sachs, who, as part of a recently-published note on the economics of the Olympics, highlighted the strength of the host-nation advantage in certain sports and used the bank’s “Growth Environment Score”, a proprietary summary statistic used to score countries’ institutional structures, as a key independent variable.
• PWC’s briefing paper Modelling Olympic Performance, which found population, average income levels (measured by GDP per capita at PPP exchange rates), ex-Soviet status and host-nation effect to be statistically-significant factors.

Here I will follow along the FT medal-tracking exercise and evaluate the model predictions against the naive forecast based on the outcomes of the 2008 Beijing Olympics. My aim is to evaluate the predictions, which is a bit different that the purpose of the FT exercise, which I take is to evaluate country performance versus expectations.

However, to evaluate country performance appropriately, one needs to know the fidelity of ones expectations. Otherwise you won;t be able to distinguish a country's outperformance from just poor expectations. Stay tuned as the games progress for the quantitative analysis.