Monday, May 14, 2012
Sport and Policy Evaluation: Case of Arsenal
To review, the start of the season started off miserably for Arsenal -- In August they took only 1 of a possible 9 points and saw their two biggest stars Samir Nasri and Cesc Fabregas depart for Manchester City and Barcelona (the former got a league title, the later did not). Following an 8-2 thrashing by Manchester United, the worst was expected for the upcoming season. Your humble blog host even predicted a 6th place finish for the squad.
It turns out that Arsenal finished 3rd, and received an automatic Champions League qualification spot.
So was the season an improvement over the previous season?
to compare relevant data, the 2012 squad earned more points (70) than the 2011 version (68). But you might protest, the 2012 squad finished a distant 19 points off of the champions, whereas the 2011 squad was only 12 points off the pace. A more sophisticated rejoinder would explain that the 2012 squad took 6.7% of all points earned in the EPL during 2011-2012 (70/1047) whereas the 2011 squad earned only 6.6% of total points earned (68/1029). Further the 2012 squad earned 7.3% of all victories in the recent campaign, whereas last year's squad earned on 7.0%.
Ah yes, but you might reply, the 2011 squad was only 4 victories off of the pace set by Manchester United, whereas the 2012 doubled that deficit against Manchester City. The 2011 squad allowed 6 less goals (43 vs. 49) but the 2012 squad scored 2 more (74 vs. 72).
This sort of game can go on for a long time (especially at a pub) with no resolution, as there are multiple measures of "good outcomes" according to different criteria.
In any evaluation, it is important to be clear -- preferably up front -- about goals to be realized and how they are measured. Goals can of course change. Every team starts out, in principle at least, with a goal to win the league, but through the season as that goal becomes unreachable (for some in the last4 minutes of added extra time, but I digress), the focus shifts to other goals, such as gaining a Champion's League spot or avoiding relegation as the case may be. This sets up the situation where there may be multiple valid metrics of evaluation, conditional on evolving experience.
So after all that on policy evaluation, did Arsenal have a better season in 2011/2012 than in 2010/2011?
The answer is an unequivocal yes.
Arsenal finished closer to the EPL title in 2012 and secured a guaranteed Champions League spot, both improvements over 2011. At the start of the season, each of these objectives would surely have been at the top of anyone's list of objectives for a top EPL squad, and will surely top Arsenal's list for next season.
So beware fancy statistics and moving goal posts. Evaluation starts with objectives, not data.