Welcome to the Science of Sport, where we bring you the second, third, and fourth level of analysis you will not find anywhere else.

Be it doping in sport, hot topics like Caster Semenya or Oscar Pistorius, or the dehydration myth, we try to translate the science behind sports and sports performance.

Consider a donation if you like what you see here!


Did you know?
We published The Runner's Body in May 2009. With an average 4.4/5 stars on Amazon.com, it has been receiving positive reviews from runners and non-runners alike.

Available for the Kindle and also in the traditional paper back. It will make a great gift for the runners you know, and helps support our work here on The Science of Sport.



Thursday, September 18, 2008

Coyle-Armstrong research installment 2

The "error" is discovered, thanks to scientific detective work

Apologies for the gap between posts - I wanted to spend more time on this particular piece to make sure that I captured what is a techical matter accurately.

In our last post, we looked at the 2005 research study published by Coyle on Lance Armstrong, in which he concluded that Armstrong's efficiency had improved by 8% over a 7 year period during which he was tested.

That finding became a crucial cog in the legal defence of Armstrong after his fifth Tour de France title, because Coyle argued that this physiological adaptation was responsible for Armstrong's dominance of the Tour. We said in our last post that there were numerous scientific flaws in the study, not least of all the fact that Coyle never once tested Armstrong during the Tour de France season. What transpired in the aftermath of the 2005 publication was only really the beginning, and those study design considerations and problems of interpretation were soon going to be overtaken by the realisation that in fact, the data was incorrect thanks to a calculation error made in working out the efficiency. That is the subject of this post.

The "minor miscalculation"

The story behind this latest revelation is that three scientists and one mathematician from Australia requested access to Coyle's data, presumably to examine it more thoroughly. At this stage, we must point out that the Coyle finding was surprising and challenged for a number of reasons:

  1. The numerous flaws in design, questions around calibration and issues of interpretation that we have already explained
  2. The fact that cycling efficiency had never been shown to change so consistently over a period of time as a result of training/maturation. Take the following quote, from Michael Ashenden, one of the paper's authors: “They were really concerned, on a scientific level, that Coyle had been able to perpetuate this myth that cycling efficiency changes."
  3. There is a theory that cancer treatment and chemotherapy, should cause a decrease in efficiency. Why? Because what happens with the removal of a testicle and subsequent chemotheraphy is that the body shifts to a greater use of fat as a source of fuel. Fat, however, is not as efficient as carbohydrate - the energy produced per liter of oxygen is lower for fat than carbohydrates, and so this theory suggests that Armstrong should, if anything, have been less efficient. This is not by itself reason for doubt - theory is, after all, any hypothesis exists in order to be proven incorrect, but it was another reason to view the Coyle data with skepticism.
However, the key point was that this "science" was being propogated as fact, when there were very clearly errors and problems with it. The scientists approach was then to ask for the data, so that they could analyse it themselves, interrogate it and discover whether in fact the conclusions were valid.

Getting the data - apparently a problem

Science is, in theory, transparent. It should be, and every scientist should be confident enough in their method and results to make data available at any time for evaluation. If they are not, then the data should not be published. That's why a scientific paper is so particular about its method - every study should be repeatable by others, to confirm or refute what it has found. If the process is legitimate, then the data should be above "reproach". It turns out that getting hold of Lance Armstrong's data was not quite so simple. Apparently, Ashenden and his colleagues were stonewalled when they raised their concerns, and eventually had to lodge a case of scientific misconduct against Coyle, with his University in Texas.

That finally got them SOME data - I must stress, that it seems they only got some of the information. If you look at the letter that Gore et al sent to JAP, they state in the opening paragraph that "Coyle made available raw data from the January 1993 test..." If you then jump to the end of the letter, you read the following: "The magnitude of this error warrants recalculation of the entire data set, but raw data from the remaining test sessions are not available from the author".

Therefore, despite having gone through an official complaint of scientific misconduct, the scientists are still not able to evaluate ALL the evidence, because its author will not make it available. Now, why is this the case? Could Coyle have "lost" the data? Highly doubtful - I suspect that given the subject and his importance, this data will be backed up numerous times. So then the only conclusion is that some reason exists not to make it available.

The Error and why the complete data set must be evaluated

The Australian group, in their letter, make the point that a calculation error had been made, which is really the catalyst for the latest round of discussion. What was that error? We gave a hint in our previous post when we introduced the following graph:
This graph shows how the delta efficiency is calculated - basically, it's the inverse of the slope of this line. So your use of oxygen rises as you do more and more work, and if you take the slope, invert it, you have a measure of how much work is doing per unit of energy consumed (you calculate energy using oxygen).

Now, what Coyle did, you'll recall, is calculate Armstrong's efficiency over 7 years from 1993 to 1999, interrupted by the cancer diagnosis and treatment.

However, what the Australian researchers realised when they looked at the data from 1993, is that Coyle had used the wrong equation. Without going into massive detail, if you look at the graph above, you will see that even when you are doing ZERO work, you are still using oxygen (otherwise, you'd be dead). This resting oxygen use, at zero load is important, because it reflects a base metabolic rate. Now, what Coyle did was to neglect this zero work point, and he FORCED the line to go through zero. This means that the slope of the relationship between oxygen use and work rate was changed because he used the wrong equation, and his calculation of delta efficiency is incorrect.

What the Australians realised is that if you used the CORRECT equation, then the values calculated change substantially. Their letter provides the numbers - they applied the correct equation and worked out that Armstrong's Delta Efficiency in 1993 was actually 23.55%, and NOT 21.75% as Coyle reported. The graph below shows the effect of this correction.

The question marks are there, of course, because we're all speculating as to how the correct equation would change the values - that data has yet to be released, so speculation is all we have...

The impact of the change

The change is huge - 8%, and therefore, the rest of the data must be evaluated. In response to this revelation, Coyle has admitted that he made an error. However, he has downplayed the importance of this error, saying that it is minor and makes "no practical difference". I might point out that his error is in fact LARGER than the change in efficiency he found in Lance Armstrong! The 8% change was significant when it was Lance's efficiency, apparently it is not when it is the error he made...

The other defence put forward by Coyle is that the error is reduced in signficance because he calculated efficiency at a high VO2, and so the effect of a resting metabolic rate is expected to be minimal. This is in fact completely incorrect. The higher the VO2, the greater the impact of the calculation on the slope. In otherwords, the slope actually changes by MORE (and hence, the efficiency changes) when you have a high VO2, than a low VO2. So Coyle's defence doesn't hold there either.

Summing up: The big picture

The relevance of this error, and the whole process of evaluating this paper, extends into the scientific community, perhaps more than it does the cyclist. So these two posts have been much more technical than we usually write, but hopefully you can appreciate the importance of discussing this kind of scientific misinterpretation and error.

In response to it all, Coyle is quoted in the New York Times as saying: “This is a minor waste on my time. However, I don’t understand how they can afford to spend so much time on this. Don’t they have real jobs?”

I suppose one would put this down to opinion, but as I wrote the other, I would say that as a scientist, your "real job" is to pursue scientific truth. So in fact, they did exactly what they were supposed to. It doesn't seem that the same applied in 2005 when the study was done.

Ross

13 Comments:

Anonymous said...

Does this data now put Armstrong at risk for further doping charges? Are his old wins' credibility shattered due to one piece of mis-interpretted data?

Anonymous said...

What about the fact that gross efficiency, which is not effected by the intercept and therefore not in "error", is still 8% higher over the course of the study?

Ward said...

Very interesting post.

I was wondering if you could point me in the direction of a citation for the general idea of: 'removal of a testicle and subsequent chemotheraphy is that the body shifts to a greater use of fat as a source of fuel'
I know that is not the main point of your argument but I am interested in the background of this thought.
thanks

Anonymous said...

I'm not quite sure I understand how a higher VO2 leads to the Y intercept being more important... I'm thinking that by higher VO2, you mean higher workload, correct? As I see it, the Y intercept could be significant or not completely depending on where it actually is and where the data points (O2, workload) lie. Because of this though, it would seem the entire data set is useless without the Y intercept. What happens to people's metabolic rate as they age through their 30's. It goes down, correct? So the Y-intercept is decreasing throughout the tests. Could we make a round-about conclusion as to increasing/decreasing delta efficiency based on gross efficiency and given that the Y intercept is more than likely decreasing through the testing period? What I'm getting at is the plausibility of change in efficiency: is it possible?
-Tom in Seattle

Anonymous said...

Hi Guys:

Listen, I read your blog pretty faithfully and I must say that it is usually refreshingly objective and, for the most part, unbiased. In fact, I refer my exercise science students to this blog on a regular basis. However, your last two posts on Ed Coyle's "minor error" read like you have your minds made up, and the vernacular used makes it look like a you have taken personal issue with Lance Armstrong and Ed Coyle. First, let me say that I do not necessary believe that Lance is innocent, and I would encourage any effort to weed-out doping athletes, if it leads to the betterment of the sport. But, just as Coyle's findings were not supported by earlier studies examining cycling efficiency, so too has been the case with Dr. Noakes' challenge to the concept of VO2max. The scientific community still considers Dr. Noakes' view to be utterly heretical. Let me also say that I admire (and subscribe to) Dr. Noakes' contentions. My point is this: we don't necessarily know the facts one way or another, but it is in the best interest of science to discuss results (and potential research errors), instead of making blunt judgments just because you don't agree with them. It is clear from your last two posts how you feel, but I would suggest that you make your posts appear more like professional commentaries and less like you are the leader of the witch hunt for Lance Armstrong.
And I know that you guys are not that closed-minded. Other than that, keep up the interesting posts. Thanks!

Anonymous said...

I have a couple of issues with your analysis so far.

First, if Coyle made the same mistake in all of his calculations then we would expect all the data to be changed about the same amount. Since gross efficiency changed (which does not require any zeroing) it would seem that Coyle's data is probably pretty good and a similar pattern would be seen if all the data were properly corrected for delta efficiency, since resting metabolism would hardly be expected to change significantly over this time – although without the raw data it is not possible to know for certain.

Second, you wrote this: "The other defence put forward by Coyle is that the error is reduced in signficance because he calculated efficiency at a high VO2, and so the effect of a resting metabolic rate is expected to be minimal. This is in fact completely incorrect. The higher the VO2, the greater the impact of the calculation on the slope. In otherwords, the slope actually changes by MORE (and hence, the efficiency changes) when you have a high VO2, than a low VO2. So Coyle's defence doesn't hold there either."

I disagree. While the slope would change some, the larger the power at the high end the smaller the slope would change if the zero power oxygen consumption moved some and this change would be similar for all similar errors. Unless you can give me an example of what you mean by this statement I will have to say I disagree and think you are wrong.

So, if the gross efficiency changes are correct for all the tests, it seems all this uproar over the delta efficiency numbers is a lot of bombast over nothing.

One last thing. Having been involved in a lawsuit or two I am simply amazed that all of Coyle's raw data wasn't made available then as part of discovery during the lawsuit Lance was involved in, along with some of the current critiques who testified for the other side. Why didn't these people ask for it then? If I were up against Coyle and this paper I would have asked for the raw data so I could check his work in view of the stakes.

Thanks

Andrew R. Coggan, Ph.D. said...

I agree with the 5th (and 2nd) anononymous poster:

1) Coyle is correct in asserting that the higher the absolute power and VO2, the less (not more, as you claim) impact constraining the Y intercept to equal zero has upon the calculated delta efficiency, and

2) focussing on how delta efficiency was calculated is somewhat myopic. According to the data presented in Coyle's paper, over time Armstrong's VO2 at a power of 400 W must have declined by ~0.3 L/min. The only really plausible explanation for such a change is an increase in muscular efficiency. Thus, unless the raw power and/or VO2 data are incorrect, precisely how delta efficiency was calculated doesn't really matter.

Anonymous said...

Agree with the last 3 or 4 posts.

Neil said...

Yeah, its been said here a couple of times: Does Coyles defence letter in the Journal, claiming his conclusions are primarily based on Gross effieciency, still hold water? Think it important that question is address cos it is part of Coyle's defence.

Then regards your comments about potential reduced effiecency due to removal of a testicle and chemotherapy: Surely there are a wealth of other potential reactions to therapy, not documented or discovered, which could oppose the process you mentioned, to decrease efficiency? eg. EPO treatment sometimes given with chemotherapy to reduce anemia (jst read some stuff, not a physiologist at all).

Neil-Cape Town

Ray said...

I have to disagree with all the anonymous posters. I think they are missing the point. The last few posts have very little to do with Lance Armstrong and doping -- it is not a witch-hunt against Lance but a harsh criticism of Coyle.

They have a lot more to do with, how Coyle's study gets so much credibility in the media, and in the courts, when it is flawed at so many levels. The finding that he used the wrong equation, just means that it was even more flawed than previously thought.

The criticism from other scientists, is that Coyle fails to meet many scientific standards, and as such, the conclusions he draws are not supported by the data he measured, even if you take for granted the data he made up.

Phil Koop said...

The bottom line is this: Coyle's original paper claims that between 1992 and 1997, Armstrong's lean/gross weight increased from 70.5/78.9 to 71.6/79.7 at the same time that work rate increased by 30W with oxygen consumption fixed at 5.0L/minute. Where did this 30W come from? Not from reduced metabolic load - his weight increased!

If you want to disprove Coyle's findings, I see two possibilities:

1) Demonstrate that the extra 30W comes from increased base metabolic efficiency rather than muscle efficiency.

2) Demonstrate that the facts reported by Coyle, as cited above, are wrong.

If it is true that Coyle has refused to release his raw data, then that is highly disconcerting - in the history of science this is almost always a bad sign. But pointing at an error in how delta efficiency is calculated does nothing to refute Coyle's basic claim.

Anonymous said...

Above, you have a graph of metabolic power (or oxygen consumption) vs. mechanical power.
The caption states that the intercept is not at zero metabolic power (VO2). That is true, but your caption states that the intercept is equal to resting metabolic rate.

That is just plain wrong and has been known since at least 1977 (the classic Gaesser & Brooks). The intercept is substantially greater than rest and the explanation for that is not very well established.

If you are trying to clear up some confusion about the difference between gross and net efficiency, that sort of error is not helping.

You also often interchange the terms work and power. Power is the rate of performing work or the rate of consuming energy.

Ross Tucker and Jonathan Dugas said...

Hi Ray, Phil, and another Anonymous poster.

First of all, thank you Ray, for pointing out succintly what I was unable to do in the whole post! You're right, and you've seen past the trees and seen the "wood", so to speak. Couldn't have (didn't) put it better myself.

Then to Phil, fair enough. On the note of proving that Coyle's data is incorrect, that would only be possible if the data was made available, which it wasn't. However, given the numerous queries raised and problems with the method, as discussed in the first post, I think there is enough there to challenge the findings. I think that is the key point - the scientific stringency is lacking.

But yes, you're right, it is a concern that the data hasn't been made available. Where I disagree is that pointing out the error (and the numerous other problems does a great deal to Coyle's claim, because it undermines the process by which that claim was generated). I tried to explain this a bit better in my most recent post (today).

The point is, science is build on the assumed foundation of credible data collection, and that was very obviously challenged, as Ray points out in his comment above.

And then finally, to the last Anonymous poster, your point on the resting metabolic rate-VO2 issue is heeded, thank you. I still contend the figure is wrong and you're over-complicating the point that is being made.

As for the matter of the power vs. work description, that's really a classic case of bringing an unnecessary level of "stringency" to the post. I realise that you'll probably dismiss me as a "quack" and a "phony", but honestly, do you think that anyone out there is reading this looking for that level of semantics, and more importantly, does it make a difference to the message. This is not a scientific journal, it is a vehicle to bring science to life, communicate it and provide people outside of the hallowed fraternity of academia with access to this kind of insight.

I only wish that the scientists in that fraternity would apply the same level of stringency to the Journals as they do to my posts. For example, you might well go and read The Journal of Applied Physiology from March 2005 and the article by one Ed Coyle. And cast this magnifying glass over it, rather than fighing an anonymous battle with an internet website which is, as my most recent post makes very clear, serving a very different purpose, one that I will not apologize for.

Ross