Monday, October 17, 2016

Anti-Doping Drug Testing Should be Evidence Based, not "Trust Us"

In anti-doping regulation, drug tests typically play a central role in the identification of athletes who have broken the rules by taking a banned performance-enhancing substance. However, as many of us will understand from experience, medical tests are not always 100% accurate - they have a failure rate. Anti-doping is no different, and we should not expect it to be.

Consequently, scientific integrity is crucial to any successful anti-doping regime. Fallibility is not a symptom of failure, but a reality to be accommodated - whether in a justice system, in a rocket launch or in a medical procedure. Science is thus central to making such systems work as we'd like them to.

A key element of scientific integrity in anti-doping is to rigorously understand, in quantifiable metrics, the rate of false positives and false negatives in drug testing.

  • A false negative test occurs when the sample (e.g., urine or blood) contains a banned substance, but is not detected. The technical term for this rate is sensitivity;
  • A false positive test occurs when the sample does not contain a banned substance, but the test shows its presence. The technical term for this rate is specificity.
False negatives and false positives in drug testing should be contrasted with getting away with doping or an improper conviction of an innocent athlete. Drug testing is usually an important part of any sanctioning process, but is not the same thing. Here are some examples:
  • Getting away with doping. Lance Armstrong famously passed hundreds of drug test, but we now know he was doping and got away with it for more than a decade;
  • Conviction of an innocent athlete. The case of Andrus Veerpalu, a Nordic skier from Estonia, is an example of an athlete accused of doping via a drug test, but whose appeal was upheld by CAS.
In both of those cases, justice ultimately prevailed. But such justice is not guaranteed. With these distinctions in mind, we can revisit issues raises in the cases of Erik Tysse and Steven Colvert, both of whom have very compelling cases of being wrongly sanctioned for doping.

In the context of the Tysse case, WADA laboratory scientists engaged in an exchange with independent scientists in the academic literature over the issue of false positives in anti-doping drug tests. That exchange is not reassuring.

In an excellent paper titled, "Can we trust anti-doping?" Waaler et al. (2011) offer a provocative thesis:
In the hunt for doped athletes, little account is taken of the risk of condemning an innocent athlete. The doping hunters appear to be more concerned with catching all the guilty than with considering the risk that some test results may be false positives.
The paper is worth reading in full, but in short, their concern is that WADA labs are not aware of the validity of the drug tests that they apply to athletes urine and blood samples. Validity refers to the quantification of the rates of false negatives and false positives in testing.

The authors conclude:
When it comes to the case in question, we must express concern: From the Adjudication Committee’s ruling it is clear that they are not certain of the facts of the case and that their decision is based on a perception that the laboratory has followed the WADA’s procedures. The chairman of the Adjudication Committee, Lars Erik Frisvold, said quite frankly during the verbal proceedings that much of the discussion that took place was over his head. Is it acceptable adherence to due process of law that the Adjudication Committee, lacking its own expertise, blindly accepts WADA’s procedures?

For doping tests, as for all other biological tests, knowledge of the validity of the test is fundamental. Validity is expressed through sensitivity and specificity. Whether sensitivity or specificity is the more important depends on the condition the test is intended to reveal. Both a"re important for doping testing. It is important to catch all cheats, but it is also important to avoid convicting innocent persons. Perhaps it is better to let ten doped athletes go free than to convict one who is innocent.

In this article we have concentrated on doping with CERA, and have had to conclude that the control system has drawn its conclusions without being able to document the validity of the test. It appears that it is assumed a priori that specificity is 100 per cent for a combination of A and B samples, which would exclude the possibility of false positive results. We have shown that lack of documentation of the validity of the test makes a conviction unreasonable and may result in a miscarriage of justice. The necessity of documenting the validity of tests applies to all doping tests – to ensure legal protection for athletes.
WADA lab scientist replied to the article in the same journal ("Doping analysis on solid ground," Hemmersbach et al. 2011), and focus on specificity, or the false positive rate, with respect to the drug test that was applied in the case of Erik Tysse (CERA):
Specificity (selectivity) is the most important validation parameter for not producing false positive results. In the period 2000 – 2007, when CERA was not yet on the market, more than 8,000 urine samples were tested at WADA laboratories in Rome, Paris and Oslo using the analytical method for detecting EPO. None of these samples had an isoelectric EPO profile that was similar to a positive CERA test, and this demonstrates the degree of specificity of the chosen method.
In other words, there has never been a false positive test. The accuracy is 100%.

Waaler and colleagues responded in a letter (translated from Norwegian via Google Translate):
Hemmersbach and colleagues write that they have infallibility in doping test work. Those responsible and involved conclude infallibility in [WADA laboratory] procedures, well not surprising. But still there is a lack an evidence of validity.
Østerud and Skotland (2012) offered another response to the WADA laboratory scientists, "Doping analyses are not on safe ground":
In their comments on the so-called Tysse affair, Hemmersbach and collaborators claim that doping analyses are on safe ground and free of error. This is a remarkable statement, in light of the fact that 94 independent experts, including 45 professors, have signed a declaration saying that data produced by the WADA laboratory in Rome fail to detect CERA in Tysse’s urine.
You can read more about the details of the Tysse case in The Edge. In addition to claiming infallibility, the WADA laboratory directors also threatened the journal which publishes the original critique of the use of science in the Tysse case.

None of this drama would be necessary if principles of scientific integrity were upheld in the WADA labs. A good step in that direction would be to require that the quantification of the validity of drug tests used in anti-doping are made available, including the details of the data and methods that underlie such conclusions. In this way the evidence base used to convict athletes would be open to scrutiny from independent experts and anti-doping regulation would be placed on a firmer foundation.

Were I an elite athlete, I'd be demanding that anti-doping regulations be evidence based. Right now, that base is on more of a "trust us" basis. Cases like those of Tysse and Colvert put that trust into jeopardy. It need not be so.

1 comment: