Critiques of Bermon and Garnier 2017 (related to Chand vs IAAF)

A new paper was published today with a strong critique of the IAAF study on testosterone and female elite athletes. 

Franklin S, Ospina Betancurt J, Camporesi S What statistical data of observational performance can tell us and what they cannot: the case of Dutee Chand v. AFI & IAAF Br J Sports Med Published Online First: 23 February 2018. doi: 10.1136/bjsports-2017-098513

That paper concludes:

we believe that it is scientifically incorrect to draw the conclusions in the Bermon and Garnier paper from the statistical results presented. Their paper claims that certain athletes have an advantage in precisely the five events where a significant effect was found: we calculate that a high share of those five significant effects are likely to be false positives.

An earlier critique was provided by statistician Andrew Gelman:

the statistical analysis data processing in this paper is such a mess that I can’t really figure out what data they are working with, what exactly they are doing, or the connection between some of their analyses and their scientific goals. 

Gelman was motivated by Simon Franklin, a post-doc at LSE, who emailed him that:

There are more than a few problems with the paper, not least the fact that it makes causal claims from correlations in a highly selective sample, and the bizarre choice of comparing averages within the highest and lowest tertiles of fT levels using a student t-test (without any other statistical tests presented).

But most problematic is the multiple hypothesis testing. The authors test for a correlation between T-levels and performance across a total of over 40 events (men and women) and find a significant correlation in 5 events, at the 5% level. They then conclude:

Female athletes with high fT levels have a significant competitive advantage over those with low fT in 400 m, 400 m hurdles, 800 m, hammer throw, and pole vault.

These are 5 events for which they found significant correlations! And we are lead to believe that there is no such advantage for any of the other events.

I also have written two critiques. First, a post-publication peer review:

My bottom line: The paper has some significant methodological issues, most notably the inclusion of female athletes who doped with those with naturally high levels of T. There is some double counting of athletes in 2011 and 2013. There is also speculation that the male findings are contaminated by doping. Methodological issues notwithstanding, the paper nonetheless strongly reinforces the 2015 CAS Chand decision. 

 And second, a short data analysis of their reported findings.

The Bermon and Garnier paper clearly has some methodological issues. However, even taken at face value it does not support the IAAF case against Dutee Chand. CAS continues to dither over the arbitration nonetheless. It is time to send it back to IAAF and ask them to start again.