Bias, Conflict of Interest and Disclosure in College Football Coaches Polls

In college football in the United States the so-called "coaches' poll" of teams has served as a key way to rank teams, with implications for bowl invitations and financial rewards. Who better, one might think, than the coaches themselves to rank teams on a weekly basis?

A new study by Matthew Kotchen and Matthew Potoski (CONFLICTS OF INTEREST DISTORT PUBLIC EVALUATIONS: EVIDENCE FROM THE TOP 25 BALLOTS OF NCAA FOOTBALL COACHES-- here in PDF and a summary at VoxEu) provides empirical evidence which shows that the expert judgments by the coaches are materially influenced by their roles as coaches and the potential for economic and other benefits that result from the rankings.

The paper concludes:

This paper provides robust evidence that private incentives have a distorting influence on the way coaches rank teams in the USA Today Coaches Poll for college football. While coaches are tasked with providing unbiased rankings of teams, they face incentives that pose potential conflicts of interest. These arise because of reputation and …nancial rewards that depend on how teams are ranked and whether teams are in position to receive an invitation to one of the high-pro…le and lucrative BCS bowl games. We …nd, based on two distinct identi…cation strategies in our statistical analysis, that conflicts of interest bias coach rankings in predictable ways.

The pattern of results shows the importance of both reputation bene…ts and direct financial payoffs. Coaches have clear incentives to rank both their own team and other teams in their athletic conference more favorably. We …find, on average, that coaches rank teams from their own conference nearly a full position more favorably and boost their own team's ranking more than two full positions. We also …find that it does not matter if a coach's team simply plays a team during the season, but coaches rank teams they defeated more favorably by more than half a position. Coaches thus make their own team look better by ranking more favorably teams they defeated.

The analysis also finds evidence that financial payoffs for the top bowl slots result in a detectable effect on the rankings of teams within their own conference, with revenues from bowl payouts typically shared across the conference.  A marginal bowl payoff of $3.3 to 5 million equates to one position higher in the rankings.

The bias that the study documents occurred during a time when the final individual coaches' polls were made public, leading to a natural question of what bias might have been in the era before disclosure. The study also provides some insight as to the effects of public disclosure on coaches rankings:

Our study is based on all of the publicly available ballots, and coaches knew their ballots would be made available when …filling them out. Does the disclosure cause them to evaluate teams differently? We do not have undisclosed ballots to make direct comparisons, but as a rough measure, we can look at the pattern of aggregate Coaches Poll results compared to computer rankings, before and after disclosure began in 2005. Our goal is to see if coaches' rankings become more similar to the computer rankings when their ballots are subject to public scrutiny.

Here is what they found:

For each team in each poll we calculate the difference between the Coaches Poll and computer rankings and average the absolute difference among observations separately for all those before and after disclosure began in 2005. . . the differences drop during both the pre and post-disclosure periods, but the drop is larger after disclosure by nearly twice as much. This implies that the difference between the Coaches Poll and the computer ranking, between two adjacent polls in the same year, is more than 100 percent smaller when the ballots are made public. It appears, therefore, that coaches are aware of their biases because their rankings move closer to the objective (computer) rankings when disclosed for public scrutiny. That coaches rankings are noticeably different under public disclosure raises further questions about how much stronger the biases are when there is no disclosure, and the result underscores the importance of public disclosure to minimize bias.

The study, as the authors note, is of relevance well beyond college football, and tells us something about the factors that shape expert judgments, when those judgments are made in the context of bias and conflicts of interest.