Sport and Policy Evaluation: Case of Arsenal

One of the valuable aspects of sport to the social scientist is that it provides a convenient laboratory for all sorts of investigations. Here I'll use the just concluded EPL season and Arsenal's campaign within it to illustrate some of the challenges of policy evaluation in any context. In the end, a good evaluation depends upon a good methodology.

To review, the start of the season started off miserably for Arsenal -- In August they took only 1 of a possible 9 points and saw their two biggest stars Samir Nasri and Cesc Fabregas depart for Manchester City and Barcelona (the former got a league title, the later did not). Following an 8-2 thrashing by Manchester United, the worst was expected for the upcoming season. Your humble blog host even predicted a 6th place finish for the squad.

It turns out that Arsenal finished 3rd, and received an automatic Champions League qualification spot.

So was the season an improvement over the previous season?

One way to answer this question would be to compare relevant data, the 2012 squad earned more points (70) than the 2011 version (68). But you might protest, the 2012 squad finished a distant 19 points off of the champions, whereas the 2011 squad was only 12 points off the pace. A more sophisticated rejoinder would explain that the 2012 squad took 6.7% of all points earned in the EPL during 2011-2012 (70/1047) whereas the 2011 squad earned only 6.6% of total points earned (68/1029). Further the 2012 squad earned 7.3% of all victories in the recent campaign, whereas last year's squad earned on 7.0%.

Ah yes, but you might reply, the 2011 squad was only 4 victories off of the pace set by Manchester United, whereas the 2012 doubled that deficit against Manchester City. The 2011 squad allowed 6 less goals (43 vs. 49) but the 2012 squad scored 2 more (74 vs. 72).

This sort of game can go on for a long time (especially at a pub) with no resolution, as there are multiple measures of "good outcomes" according to different criteria.

In any evaluation, it is important to be clear -- preferably up front -- about goals to be realized and how they are measured. Goals can of course change. Every team starts out, in principle at least, with a goal to win the league, but through the season as that goal becomes unreachable (for some in the last4 minutes of added extra time, but I digress), the focus shifts to other goals, such as gaining a Champion's League spot or avoiding relegation as the case may be. This sets up the situation where there may be multiple valid metrics of evaluation, conditional on evolving experience.

http://images.teamtalk.com/12/05/402x210/arsenal11024_2764401.jpgBut all evaluations are conditional and value laden. Sometimes we'll agree on what those conditions and values are and sometimes we won't. Reaching such agreement is much easier if there are some expectations set before experience occurs. This is why legislation and policy proposals are often used as the basis for conducting a policy evaluation, as such events mark a stabilization (even if only temporary) on expectations to be realized in practice.

So after all that on policy evaluation, did Arsenal have a better season in 2011/2012 than in 2010/2011?

The answer is an unequivocal yes.

Arsenal finished closer to the EPL title in 2012 and secured a guaranteed Champions League spot, both improvements over 2011. At the start of the season, each of these objectives would surely have been at the top of anyone's list of objectives for a top EPL squad, and will surely top Arsenal's list for next season.

So beware fancy statistics and moving goal posts. Evaluation starts with objectives, not data.