Quantifying the Value of a Football Manager

In today's FT Magazine Simon Kuper has an essay on the evaluation of football managers in England's professional leagues based on the work of his Soccernomics co-author Stefan Szymanski, of the University of Michigan.

Here how Kuper describes Syzmanski's methodology:

[F]ootball does a bad job of valuing managers. Football managers are modern celebrities, yet the vast majority appear to add no value to their teams, and could probably be replaced by their secretaries or stuffed teddy bears without anyone noticing. Only a few managers, such as Sturrock and Alex Ferguson, consistently improve their teams. Yet some of these excellent few get overlooked. All these findings emerge from a 37-year study of English football managers...

If players’ wages determine results, it follows that everything else – including the manager – is just noise. Most managers are not very relevant. In the long run, they will achieve almost exactly the league positions that their players’ wages would predict.

Still, there is an important caveat. Players’ wages don’t explain everything – merely almost everything at most clubs. That leaves room for a few good managers to make a difference. The question then is: which managers finish consistently higher with their teams than you would expect given their wage bills? Or, to borrow a phrase from Real Madrid’s manager José Mourinho, who are the special ones?

Szymanski has tried to answer that question. He analysed the financial accounts of four-fifths of English professional clubs from 1973 to 2010, and identified the managers who consistently overachieved. These men are the elite.

We should note right away that Szymanski’s model gives more credit to overachieving managers at the top of football than at the bottom. England’s 92 professional teams are spread over four divisions. A manager in League Two who has the 90th smallest budget in England but manages to finish 80th nationwide is overachieving. However, a manager with the third-highest budget in England who wins the Premier League is probably overachieving even more.

Syzmanski looks at team performance (focused on 251 managers out of 699 total who held the position for 5 years or more and for which financial data was available for the period 1974 to 2010),  after accounting for the expected position given the wages, which exhibit a strong correlation with league table position, as shown in the graph below from UEFA (here in PDF).

After accounting for wages, Szymanski then attributes the balance of unexplained variance in final league position to the "manager effect," an assumption with shortcomings that Szymanski is well aware of -- Kuper explains "factors besides the manager might have caused each club’s overachievement."

The attribution of unexplained variance to the "manager effect" is a serious weakness in such studies.  This can been demonstrated quantitatively by looking at another recent effort to quantify the value added by the manager.

Bell at al. (2011) attempt to evaluate managers in a similar fashion to Szymanski, using less data and a more complex statistical methodology. Their paper, titled, "The Performance of Football Club Managers: Skill or Luck?" evaluates managerial performance 2004 to 2009 in the Premier League at the match level and account for a range of variables, such as injuries, suspensions, transfer spending . They find, as did Szymanski, that weekly wages alone explains more than half the variance in points awarded (56% to be exact).

But then they do something that I don't quite follow -- they create a complex multiple regression model that throws in a suite of variables (some of which are not statistically significant) and end up explaining only 20% of the variance (see their Table 1, p. 21), which represents a severe degradation from the simple bivariate model. They then attribute the remained 80% of unexplained variance to the "manager effect." If I've understood their methods correctly, this simply seems implausible.  (I've emailed the authors and they are welcome to correct any misinterpretation.)

Despite the different approaches, the results of Bell at al. and Szymanski are similar -- for instance, the best managers in the Premier League appearing on both lists are Ferguson, Wegner, Beneitez, Moyes.  Each of these managers has led teams with unusual success (Everton was top 6 in 3 of the 5 years looked at by Bell et al. and won promotion with Preston North End in the longer period looked at by Szymanski). So yes, these teams outperformed, with either Manchester United or Arsenal winning the Premier League championship in 15 of 20 seasons.

But were these teams successful because of the manager?  Or were the managers successful because of the teams? What if it was both? And more? The only way that these studies can answer this question is by assuming the role of the manager in the variance to be explained, which is, unfortunately, the exact relationship that these studies are trying to pin down. So if you buy the assumptions of how to attribute the "unexplained variance" then these studies provide an answer. But if you don't buy the assumptions, then you wind up right where you started.

Consequently, I am not yet convinced that anyone has solved the riddle of effectively quantifying the value added by a manager, though the efforts by Syzmanski and Bell et al. represent a good start.