Was the PFPO as good prediction? Well, on the one hand correctly picking the winners of 11 of 14 games up to the final seems pretty good. But as (the few) readers here know, evaluating a prediction based solely on relative accuracy can be misleading. It is important to ask if the prediction had skill. meaning an improvement on a naive baseline expectation.download the pdf). If Barcelona wins the trophy, we will reach the threshold of 80% correct predictions. However, the gap between the Catalan team and Manchester United is negligible. All will depend on their form on the day, injuries and, of course, chance!
Our prediction model confers a relative strength to clubs on the basis of four criteria: the experience accumulated by players in Champions League matches, their quality in terms of participation in recent international matches, weighted according to the level of the national team represented, the teams’ squad stability, and the results obtained in the group stage of this year’s Champions League competition. The club with the richest squad has an index of 100. All other values have been adjusted according to it.
UEFA club coefficients, calculated during each season and used to seed teams in the Champions and Europa League draws. They are publicly available and thus readily used as a naive baseline.
How would a prediction of last season's knock-out stages of the Champions League have looked using the UEFA team coefficients for 2010/2011 at the start of the knock-out stage?
The higher ranked team according to the UEFA team coefficients at the start of the knock-out stage would have correctly predicted 13 out of 14 games prior to the final (missing out on Inter). This means that the PFPO methodology did not perform as well as the naive methodology -- that is, it showed no skill compared to a naive baseline expectation. A small consolation for the PFPO is that it predicted Barcelona to win the Champions League where as the UEFA coefficients gave it narrowly to Manchester United.
I'll re-run this exercise later in the season when we get to the round of 16. Stay tuned.