Welcome to the Science of Sport, where we bring you the second, third, and fourth level of analysis you will not find anywhere else.

Be it doping in sport, hot topics like Caster Semenya or Oscar Pistorius, or the dehydration myth, we try to translate the science behind sports and sports performance.

Consider a donation if you like what you see here!


Did you know?
We published The Runner's Body in May 2009. With an average 4.4/5 stars on Amazon.com, it has been receiving positive reviews from runners and non-runners alike.

Available for the Kindle and also in the traditional paper back. It will make a great gift for the runners you know, and helps support our work here on The Science of Sport.



Monday, September 15, 2008

Coyle and Armstrong: Research "errors" evaluation

The Coyle study on Armstrong: A "minor error" or a scientific "hoax?" Analysis and insight

As promised, we turn our attention to this story, which broke last week, co-inciding with the news that Lance Armstrong is coming out of retirement and will try to race the 2009 Tour de France. It's quite an intricate story, and technical, so forgive the longish post, but we try to go back to the beginning and then work through the sequence of events in order.

The story, which was reported in the New York Times, reports that Ed Coyle, physiologist at the University of Texas, admitted to making what he calls a "minor error" when calculating Lance Armstrong's efficiency during his research. That "minor error" happens to have a major impact on the study's findings, since his main finding was that Armstrong became more efficient between 1993 and 1999.

The paper was, it must be said, widely criticized from the beginning. It drew two separate letters, criticizing the methods, debating the scientific stringency of testing, and questioning the conclusions. It became something of a "shining light" to research without quality control, a running joke of sorts within sections of the scientific community.

I recall attending a conference in the USA soon after its publication - it was the hot topic, of course, because not only was Coyle doing research on an elite subject, it was THE elite subject - the "greatest physiological specimen in the world". Just like roadies wait years to see rock stars or musicians, any exercise physiologist would leap at the chance to publish data on a record-breaking Tour de France cyclist!

So predictably, the paper was something of a conversation starter at scientific conferences and in the media. At the conferences, conversation was not positive, however, with many dismissing it as trivia, rather than science. They were being kind...Few would have expected the next two years to keep the paper quite as much in the public and legal eye, since it became a legal defence vehicle for Armstrong, as we shall see...

With regards to the media, the paper was huge - it was reported widely as "fact" that Armstrong's success was the result of his never-seen-before increase in efficiency. This is typical of how media spin sometimes contaminates science. In this particular case, that science was not even particularly "clean", with many holes, but nevertheless, the media lapped it up. This is of course frustrating for most scientists, since the "sensationalization" of science is rarely constructive. When it is also poor science, all the more reason to pursue the truth...

First things first: The Coyle study from 2005. What was found?

To begin with, we have to look back and report on the findings of Coyle in the research study that is now the focal point of the "error".

The paper was called "Improved muscular efficiency displayed as Tour de France champion matures", which kind of reveals the paper's hand from the very first line. Here's a breakdown of what Coyle did (note that we're focusing only on the efficiency part, and not some of the other measurements made. If interested, you can download the entire paper here. See also the end of this post for the links to the entire series of exchanges in the Journal of Applied Physiology.)

The figure below demonstrates just what Coyle did, and what he found.


The research began in November 1992, when Coyle did his first battery of tests on Armstrong. He then did a second test a few months later, followed by a third in 1993. Then a long break interrupted the testing, and it was in that period that Armstrong was diagnosed with and treated for testicular cancer.

Testing resumed in August of 1997, and the final test took place in November 1999. This was the only testing session that co-incided with Armstrong's Tour de France dominance (1999 to 2005), although it must be pointed out that it was done in November, four months AFTER Armstrong won the 1999 Tour. This has relevance for Coyle's conclusions, as we shall see.

The test: Explaining efficiency measures

Testing consisted of a VO2max test, during which time, Coyle measured gases (oxygen in, CO2 out) and did a blood lactate measurement at the end of the test. He calculated two important variables:

  • Gross Efficiency - the ratio of work done to energy expended to do the work. The work done is taken from the power output, while the energy expended to do the work is calculated using the respiratory gases and calculations we won't get into here. But for example, if a cyclist is riding along at 200 W, and their respiratory gases are used to calculate that their energy consumption is 1000 W (or Joules per second), then that cyclist is 20% efficient, according to this method.
  • Delta Efficiency - this is a more comprehensive method, because it is calculated as the ratio of the change in work done per minute to the change in energy expended per minute. It is considered a better measure of efficiency because it takes into account the use of oxygen (and energy) at rest and when no work is being performed.
This gets technical, but the simple way to think of this is that as you do more work, your oxygen use rises. We can use that oxygen use to calculate how much energy you are using, and then say that energy use is proportional to work rate (that's fairly obvious, hopefully).

Now, if we take the inverse of the slope of that line (in otherwords, work done vs. energy use), then we can work out delta efficiency. However, it's critical that this slope take into account what the energy use was when you were not doing any work - the resting energy use, and also the energy use when cycling at zero load. The graph below is schematic, but I use it illustrate the point - the energy use rises with increasing work rate, but must take into account energy use when work rate is zero.


Studies as far back as 1975 (Gaesser & Brooks) have shown that gross efficiency tends to skew the results, because of the failure to account for energy use at zero load. Therefore, delta efficiency is considered the better method, but only if used properly, as we'll see!

Coyle's study found that Armstrong's efficiency increased progressively over the 7 years in which he was tested, as shown in the figure above. His delta efficiency improved from 21.37% in 1992 to 23.12% in 1999. This increase (1.8 percentage points) is relatively small, and it must be noted, is actually less than the typical error of the equipment used to measure it with! In other words, taking nothing but equipment variation into account, this kind of change is possible...

Based on this finding, Coyle named his study, and the theory was published that Lance Armstrong had seen a progressive increase in his efficiency over the years. This was to become a key part of this "armour" in 2005, when this study would provide some support for his claims that his ascendancy in the world of cycling was "natural". Coyle speculated on a number of physiological factors explaining this finding (changes in enzyme activity, muscle fibre switches etc.). However, the first responses to Coyle's paper were swift...

The first response: Criticism of methods and overinterpretation of data

The first response and criticism was swift, and came from two sources. First, David Martin and colleagues from Australia wrote a letter titled "Has Armstrong's cycle efficiency improved?". This was accompanied by a letter from Yorck Olaf Schumacher and his colleagues titled "Scientific considerations for physiological evaluations of elite athletes".

Essentially, these letters criticized the study design, the method and the scientific process follwed, including the conclusions. The raised the following points:
  • Timing of testing sessions - Coyle very clearly concluded that his measurements of muscular efficiency were of paramount significance to Armstrong's Tour victories. He denies this, but the title and his conclusions make very clear that his view is that Armstrong's success is a function of this improved efficiency. Coyle would go on to testify in court that Armstrong's rise could have been achieved without doping, so it's quite clear that his finding was intended for support of Armstrong's Tour performance. Yet remarkably, NOT A SINGLE testing session co-incided with the Tour. All the testing happened out of season, and only the 1999 test even overlapped with the Armstrong Tour victories.
  • Issues around equipment - calibration, reliability, validity etc., which we won't get into here, other than to say that over a period of seven years, the control of equipment is obviously crucial. Coyle responded to these queries, and they do not seem to have huge influence over the current debate
  • The conclusion - Coyle's was really one of the first papers to even suggest that muscular efficiency improves over time and with training. While this would seem intriguing, it also disagrees with many other findings, which are that extensive endurance training does not improve cycling efficiency. Also, efficiency is not a factor that seems to be associated with performance in elite cyclists, and so the conclusions are 'liberal', to say the least.
The next steps: Re-analyzing the data and digging up errors

These issues are primarily behind my earlier observation that the paper was widely criticized, even early on. However, what transpired next is even more significant, because Christopher Gore, Michael Ashenden, Ken Sharpe and David Martin continued their quest for the "truth", and eventually managed to get hold of (some) data from Coyle's testing.

Between the publication of the paper in 2005 and the latest round of debate, there was also the matter of a court case in which Coyle was a paid expert witness on behalf of Armstrong. His testimony was aimed at building a credible case for how Armstrong could have dominated the sport for 7 years thanks to the remarkable physiology put forward in this paper. And so this study, with its holes, flaws and inaccuracies, actually went on to form part of a legal argument despite those problems. It also reveals a big part of Coyle's incentives, something we'll look at in our next post.

These holes and flaws in the Coyle study however pale into insignificance when compared to the latest revelations, where analysis, and some "between the lines" reading of Coyle's data revealed outright errors in the research. That is, it's no longer a case of questionable methods and over-interpretations, it's now a matter of miscalculation and wrong results. All the way from the lab, into the media, and on into the court-room!

But in the name of time (and length!) I'm going to call it for today's post, and leave you with that teaser, which we'll pick up on tomorrow, when we look the "minor error" and what impact it has on the results.

Join us then!

Ross

Links to the original article and follow-up letters:

12 Comments:

Anonymous said...

Outstanding blog today, Ross!! This is a very important and complicated story, with many highly technical aspects and the possibilities of flawed research and even deception. You have explained it better than anyone else ever could. Can't wait for tomorrow's installment!

Ross Tucker and Jonathan Dugas said...

Hi Owen

Thanks a lot! Glad you enjoyed it - difficult to write, I'm still not 100% on what had to be left out vs. included, so I'm pleased you enjoyed it.

Will still make contact with you about that interview/collaboration, I haven't forgotten about it!

Thanks a lot!
Ross

Unknown said...

I've read the original Coyle article and there are many interesting things in it and I look forward to your interpretation of the data. From doing my own quick calculations and then comparing it to the data table in the article- he stayed at a power to weight ratio of ~5 watts/kg in an untrained state across the board with the exception of the first year measured where he was 4.75 watts/kg. What is interesting about this is the Coyle is extrapolating his data to include race weights during the Tour that he did not measure- as well as assumed values for Vo2 max during this period, also not measured and then comparing it with data from 91-95 that was actual! I can see where Coyle estimates the 8% improvement in the delta efficiency, but where is the math coming from for him to conclude an 18% improvement? It doesn't wash.
I look forward to tomorrows post!

Ross Tucker and Jonathan Dugas said...

Hi Jen

The "assumption" of Lance's weight is one of the most incredible "luxury" assumptions I've seen in a scientific paper, and you're right, it's a big part of the argument.

That was picked up on in David Walsh's book, and the study makes it perfectly clear that Lance didn't return from cancer having lost 10 kg as the lay media so often report. Of course, he was never tested during competition, which is a big problem with the study as well, as we mentioned. But still, to just assume that his racing weight was 72kg is a giant leap of faith for a serious scientific paper. Let's all just make up data from now on...

As for the 18%, this comes from the assumption of weight. You will have seen that Coyle measured his power output at a VO2 of 5.0L/min, and it went from 374 to 403 W. That means an 8% increase. Now, to get this to 18%, you have to believe that his weight also went down from 79kg to 72kg. So that 374W is equal to 4.74W/kg, and the 403W is equal to 5.60W/kg (you have to believe the new weight, it's incredible). That's an 18% increase.

Just astonishing use of liberty...

Ross

Anonymous said...

Guy's remember reading this paper whhen it came out - but not my area of expertise now - has anyone else published on muscular effeciency improving - presumably if it happened Asker Jeukendrup or Louis Passfield would have mentioned it somewhere - not to mention GB cycling!

Danny M said...

I remember studying this paper my 1st year in grad school.
Very interesting what has came about.

You did a great job explaining it Ross?
I have a couple of questions but you'll probably get them on the the next post!
Thanks for bringing the topic up again.

Anonymous said...

Spellbinding stuff! You've got me hooked..looking forward to the conclusion of this story, and hopefully some conclusive proof of LAs suspicious past.

Anonymous said...

Groupies eagerly wait for rock stars. Roadies are the jaded road crew who have no illusions about the show because they see it everyday.

Unknown said...

For a cyclist, unlike an endurance runner, the 3 most important factors in maintaining speed on a cycle are (1)the ABSOLUTE power that can be maintained, not the relative (per body mass), (2) reducing air drag, and (3) changes in cycle technology. Because Lance Armstrong was tested in conditions where (2) and (3) did not change, then (1) absolute power is most important.

Power at O2 uptake of 5.0 l/min, in Watts (Table 2 of the original paper 374 (at age 21.1), 382 (21.4), 399 (25.9), 404 (age 28.2). Lance Armstrong was able to accomplish at a VO2 he would easily be able to maintain during a race, that was 30 watts higher at age 28 vs. 21. This is a SUBSTANTIAL increase in power production (8%) that could translate into faster cycling times maintained over a long Tour de France stage.


It is important to remember that both GROSS and DELTA Efficiency are Calculated from these values, and thus are not more important in evaluating or predicting Lance Armstrong's performance than the raw power values. Any correction for the intercept of the Delta Efficiency values (one of the controversial points raised in letters to the editor of the Journal of Applied Physiology) would have NO mathematical influence on the slope of the line or the change in Delta efficiency.



John Lawler, PhD

Anonymous said...

One thing nobody mentions is power values from VO2 max and submaximal.

Lance's VO2 max was about 5.9 liters per minute. A good really rider can only sustain about 80% of VO2 max for 1- hour sustained power. So thats about 380-390 watts. If he produces 403 watts at 5.1 liters as stated.

Your maximal aerobic power is always the ceiling for your sustainable power. Anyway, 'Michele Ferrari' has Lance sustaining about 6.7 watts per kilo for 30 minutes on some climbs. So thats a 1-hour sustained power of maybe 6.5. So we are talking in the region of 466-500 watts for sustainable power all of a sudden. I dont think that using Lance's highest ever VO2 max test is a bad baseline for what he can produce. VO2 max power is usually 5-minute power output. So Lance's 5 minute power ouput suddenly becomes his 60 minute power output.

Anonymous said...

Surely a reasonable person must conclude that there is only a tiny chance that Lance Armstrong didn`t use performance enhancing drugs in order to win the Tdf.

The topic may be interesting but personally I find it sad that you let the guy off the hook instead of saying : CHEATS EFF OFF.
History will not judge LA kindly - of that I am sure.

Albert Lörenz
Stuttgart, Germany.

(nb. Jan Ulrich is treated largely with contempt here these days because we all know he is/was a cheat. Pity the Anglo/Saxon sporting world doesn`t have the same standards...)

Ross Tucker and Jonathan Dugas said...

HI Albert

Thanks for the post. Succinct and to the point, but I agree with you - I guess our definition of "reasonable" might differ from those who are so hostile above.

INteresting about Ullrich. As for the AngloSaxon standards, I suspect a line can be drawn somewhere through the Atlantic to define the general differences in perception!

Only general though...

Ross