Here how Kuper describes Syzmanski's methodology:
Syzmanski looks at team performance (focused on 251 managers out of 699 total who held the position for 5 years or more and for which financial data was available for the period 1974 to 2010), after accounting for the expected position given the wages, which exhibit a strong correlation with league table position, as shown in the graph below from UEFA (here in PDF).
If players’ wages determine results, it follows that everything else – including the manager – is just noise. Most managers are not very relevant. In the long run, they will achieve almost exactly the league positions that their players’ wages would predict.
Still, there is an important caveat. Players’ wages don’t explain everything – merely almost everything at most clubs. That leaves room for a few good managers to make a difference. The question then is: which managers finish consistently higher with their teams than you would expect given their wage bills? Or, to borrow a phrase from Real Madrid’s manager José Mourinho, who are the special ones?
We should note right away that Szymanski’s model gives more credit to overachieving managers at the top of football than at the bottom. England’s 92 professional teams are spread over four divisions. A manager in League Two who has the 90th smallest budget in England but manages to finish 80th nationwide is overachieving. However, a manager with the third-highest budget in England who wins the Premier League is probably overachieving even more.
The attribution of unexplained variance to the "manager effect" is a serious weakness in such studies. This can been demonstrated quantitatively by looking at another recent effort to quantify the value added by the manager.
Bell at al. (2011) attempt to evaluate managers in a similar fashion to Szymanski, using less data and a more complex statistical methodology. Their paper, titled, "The Performance of Football Club Managers: Skill or Luck?" evaluates managerial performance 2004 to 2009 in the Premier League at the match level and account for a range of variables, such as injuries, suspensions, transfer spending . They find, as did Szymanski, that weekly wages alone explains more than half the variance in points awarded (56% to be exact).
But then they do something that I don't quite follow -- they create a complex multiple regression model that throws in a suite of variables (some of which are not statistically significant) and end up explaining only 20% of the variance (see their Table 1, p. 21), which represents a severe degradation from the simple bivariate model. They then attribute the remained 80% of unexplained variance to the "manager effect." If I've understood their methods correctly, this simply seems implausible. (I've emailed the authors and they are welcome to correct any misinterpretation.)
But were these teams successful because of the manager? Or were the managers successful because of the teams? What if it was both? And more? The only way that these studies can answer this question is by assuming the role of the manager in the variance to be explained, which is, unfortunately, the exact relationship that these studies are trying to pin down. So if you buy the assumptions of how to attribute the "unexplained variance" then these studies provide an answer. But if you don't buy the assumptions, then you wind up right where you started.
Consequently, I am not yet convinced that anyone has solved the riddle of effectively quantifying the value added by a manager, though the efforts by Syzmanski and Bell et al. represent a good start.