Available for the Kindle and also in the traditional paper back. It will make a great gift for the runners you know, and helps support our work here on The Science of Sport.
Boston "not significantly aided"? The ARRS stastician view and its faults
Boston was "not excessively aided" say the ARRS. But their conclusion has (major) problems...
We hope everyone is enjoying a timely Easter break. Having wrapped up Boston in my previous post, I received a few emails and comments about another analysis of Boston, and thought I'd comment briefly on it here rather than in the discussion thread to previous posts or on Twitter!
So the following article was released today, an excerpt of which appears below, under the headline "Published Analysis shows Mutai's Boston time not excessively aided"
Very quickly, some thoughts on this to fill the Easter weekend break!
Using ranking to justify times
First off, the astonishing statement is that Mutai would be expected to significant break the record because he was ranked first in the world, compared to Gebrselassie who was ranked tenth when he broke 2:04. Well, perhaps they've forgotten that Gebrselassie was already the world record holder. And that was only one year earlier! So they can take that 10th ranking and lob it into a trash can, because he was only 26 seconds off the 2:04 barrier when he stood on the start-line, a year earlier, and therefore was totally expected to have a shot at beating the time. Which he did, by 29 seconds.
Now, they're suggesting that a man ranked number 1 (and make no mistake, Geoffrey Mutai is pedigreed, and has "WR holder" written all over him), should break a record by 58 seconds given that his previous best was 56 seconds off the old one, but that he was ranked # 1...
That's poor inference if ever I have read it.
The next big claim in the article is that they estimate an improvement of 1:37 which they suggest is not excessively aided. Granted, that's their estimate of how much Mutai benefited, and one can argue that he naturally achieved this leap in performance (I'd argue very strongly against this), but there are a couple of issues.
First, the overall conclusion (and heading of the article) is flawed because 1:37 is a big difference to make if that is their estimate. To illustrate, it takes the time from being greatest ever by almost a minute, to making Mutai only 6th fastest performer in history, so I don't know how they arrived at "not excessively aided"...it's a huge difference.
The thing about that is they've set what is a very poor (and possibly arbitrary) cut off for "excessively aided" at 5 sec/km. I am not sure why they've chosen this size, but when was the last time we saw the marathon world record bettered by 3:30? (I can tell you the answer - it was Jim Peters in 1952).
And if we did see a WR beaten by 3:30, what would our reaction be? "Measure the course again", I suspect! Imagine the 10k WR going from 26:17 to 25:27 in one run - we would think the guys had miscounted the number of laps and done 24 rather than 25! The 5,000m record improved by 25 seconds? When Gebrselassie broke it by 11 in 1995 is was staggering.
Now granted, I'm not sure whether the ARRS would apply their 5 sec/km principle to these events, but I'm illustrating the point - in a competitive sport with long history and access, performances at the level of a world record, in any event, are not improved by that kind of margin, ever. The marathon world record will not fall by more than 3 minutes. (Of course, it's possible that the record in some trail run or small event is broken by the amount, but the key is the "strength and depth" of the event, which is a function of how many people do it, over how long and what quality they are. The marathon is solid, it's too strong and too old for "weak records")
So while interesting, I think they have lost sight of the wood for the trees. The first mistake is to look at the depth of the field, when really, the issue is the top, top level of runner. The first five to ten men tell you what really happened, and they are the men who DO NOT improve by 5 seconds/km when already at the WR limit. As I said, the WR in the marathon has not leaped forward by 5 sec/km in almost 60 years.
In fact, since Dinsamo's record of 2:06:50 was bettered in 1998, SIX world records have been set, and the AVERAGE improvement has been 28.3 seconds. That's 0.7 sec/km. So a 5 sec/km margin might be fine for the masses, or even the near-elite, but for men trying to run faster than any human in history, 5 seconds per kilometer is, frankly, ridiculous. And that's why a 1:37 "aiding" is massive!
As mentioned, this is the result of the method they use, which I don't think is very sound to answer this question. That is, don't look at 43 elites, look at the very best in relation to the very best in previous years. So take a look at point 7 in this Letsrun.com recap of the week (the whole article is worth reading, in fact), because it presents that the wind had a very meaningful effect. A record that stood for many years (from 1994) is bettered by Robert Cheruiyot, then bettered again in 2010, and then suddenly, seven men are faster than that, five of them by more than a minute. That's excessive, in performance language, no matter what "stats" suggest.
This is a classic case of whether statistical significance is more relevant than what is meaningfully different. A lot of times in sports science research, studies will use statistical methods to conclude that a treatment or method (say for example altitude training) is not beneficial because it results in an improvement of say 15 seconds over 5,000m compared to a control group that for example did sea-level training. And sure, there's a chance that these 15 seconds are down to 'chance' and thus not significant. But equally, any elite athlete will tell you that "IF" they can find a 15 second improvement, they'll take it. Most athletes will take an improvement of 5 seconds over 5,000m at the level of Olympic contender, because that's the difference between podium and back of the pack!
So there's a new method of statistics for sports science studies (and others, of course) where you now look for meaningful differences between groups. And I can pretty much guarantee that a 1:37 improvement for an athlete who was already at 2:05 is going to be a meaningful difference. Was it just a great day? Was it the wind? I've expressed my view that at that level, the margins are seconds, not minutes, and so I think the wind had a big effect.
However, as we've said before, Berlin and Chicago will tell us, because if Mutai and Mosop are in the same shape there, well, let's expect at 2:03:00 again...but until then, I believe the statisticians are missing something!
Enjoy Easter!
Ross
We hope everyone is enjoying a timely Easter break. Having wrapped up Boston in my previous post, I received a few emails and comments about another analysis of Boston, and thought I'd comment briefly on it here rather than in the discussion thread to previous posts or on Twitter!
So the following article was released today, an excerpt of which appears below, under the headline "Published Analysis shows Mutai's Boston time not excessively aided"
Writing in this week's "Analytical Distance Runner," the official ARRS publication, chief analyst Ken Young wrote: "It should also be noted that Geoffrey Mutai (KEN) who ran 2:03:02 (in Boston), was ranked #2 on the ARRS competitive rankings and he had been ranked #1 for four weeks earlier in the year. When Haile Gebrselassie (ETH) set the current world record of 2:03:58.2, he was ranked #10 in the world. Hence, it is not surprising that Mutai could be capable of significantly bettering the world record. Too bad that he ran this on a course that is not eligible for records."So they are estimating an improvement of 1:37 for the men, which is then apparently not significantly aided, this is judged according to criteria that anything improved by more than 5 seconds/km is considered excessively aided.
Very quickly, some thoughts on this to fill the Easter weekend break!
Using ranking to justify times
First off, the astonishing statement is that Mutai would be expected to significant break the record because he was ranked first in the world, compared to Gebrselassie who was ranked tenth when he broke 2:04. Well, perhaps they've forgotten that Gebrselassie was already the world record holder. And that was only one year earlier! So they can take that 10th ranking and lob it into a trash can, because he was only 26 seconds off the 2:04 barrier when he stood on the start-line, a year earlier, and therefore was totally expected to have a shot at beating the time. Which he did, by 29 seconds.
Now, they're suggesting that a man ranked number 1 (and make no mistake, Geoffrey Mutai is pedigreed, and has "WR holder" written all over him), should break a record by 58 seconds given that his previous best was 56 seconds off the old one, but that he was ranked # 1...
That's poor inference if ever I have read it.
"Not excessively aided"? The 5 sec/km implication
The next big claim in the article is that they estimate an improvement of 1:37 which they suggest is not excessively aided. Granted, that's their estimate of how much Mutai benefited, and one can argue that he naturally achieved this leap in performance (I'd argue very strongly against this), but there are a couple of issues.
First, the overall conclusion (and heading of the article) is flawed because 1:37 is a big difference to make if that is their estimate. To illustrate, it takes the time from being greatest ever by almost a minute, to making Mutai only 6th fastest performer in history, so I don't know how they arrived at "not excessively aided"...it's a huge difference.
The thing about that is they've set what is a very poor (and possibly arbitrary) cut off for "excessively aided" at 5 sec/km. I am not sure why they've chosen this size, but when was the last time we saw the marathon world record bettered by 3:30? (I can tell you the answer - it was Jim Peters in 1952).
And if we did see a WR beaten by 3:30, what would our reaction be? "Measure the course again", I suspect! Imagine the 10k WR going from 26:17 to 25:27 in one run - we would think the guys had miscounted the number of laps and done 24 rather than 25! The 5,000m record improved by 25 seconds? When Gebrselassie broke it by 11 in 1995 is was staggering.
Now granted, I'm not sure whether the ARRS would apply their 5 sec/km principle to these events, but I'm illustrating the point - in a competitive sport with long history and access, performances at the level of a world record, in any event, are not improved by that kind of margin, ever. The marathon world record will not fall by more than 3 minutes. (Of course, it's possible that the record in some trail run or small event is broken by the amount, but the key is the "strength and depth" of the event, which is a function of how many people do it, over how long and what quality they are. The marathon is solid, it's too strong and too old for "weak records")
So while interesting, I think they have lost sight of the wood for the trees. The first mistake is to look at the depth of the field, when really, the issue is the top, top level of runner. The first five to ten men tell you what really happened, and they are the men who DO NOT improve by 5 seconds/km when already at the WR limit. As I said, the WR in the marathon has not leaped forward by 5 sec/km in almost 60 years.
In fact, since Dinsamo's record of 2:06:50 was bettered in 1998, SIX world records have been set, and the AVERAGE improvement has been 28.3 seconds. That's 0.7 sec/km. So a 5 sec/km margin might be fine for the masses, or even the near-elite, but for men trying to run faster than any human in history, 5 seconds per kilometer is, frankly, ridiculous. And that's why a 1:37 "aiding" is massive!
As mentioned, this is the result of the method they use, which I don't think is very sound to answer this question. That is, don't look at 43 elites, look at the very best in relation to the very best in previous years. So take a look at point 7 in this Letsrun.com recap of the week (the whole article is worth reading, in fact), because it presents that the wind had a very meaningful effect. A record that stood for many years (from 1994) is bettered by Robert Cheruiyot, then bettered again in 2010, and then suddenly, seven men are faster than that, five of them by more than a minute. That's excessive, in performance language, no matter what "stats" suggest.
The issue with statistics and meaningful differences
This is a classic case of whether statistical significance is more relevant than what is meaningfully different. A lot of times in sports science research, studies will use statistical methods to conclude that a treatment or method (say for example altitude training) is not beneficial because it results in an improvement of say 15 seconds over 5,000m compared to a control group that for example did sea-level training. And sure, there's a chance that these 15 seconds are down to 'chance' and thus not significant. But equally, any elite athlete will tell you that "IF" they can find a 15 second improvement, they'll take it. Most athletes will take an improvement of 5 seconds over 5,000m at the level of Olympic contender, because that's the difference between podium and back of the pack!
So there's a new method of statistics for sports science studies (and others, of course) where you now look for meaningful differences between groups. And I can pretty much guarantee that a 1:37 improvement for an athlete who was already at 2:05 is going to be a meaningful difference. Was it just a great day? Was it the wind? I've expressed my view that at that level, the margins are seconds, not minutes, and so I think the wind had a big effect.
However, as we've said before, Berlin and Chicago will tell us, because if Mutai and Mosop are in the same shape there, well, let's expect at 2:03:00 again...but until then, I believe the statisticians are missing something!
Enjoy Easter!
Ross
0 Comments:
Post a Comment