Pistorius and CAS: Incentive clash

How much does it cost to buy a scientific opinion? Science for sale?

Last week, in Lausanne, at the Court of Arbitration hearing into the legality of Paralympic athlete Oscar Pistorius' high-tech carbon fibre blades, Professor Hugh Herr of the Massachusetts Institute of Technology (MIT), provided scientific evidence to support the Pistorius defence claim that:

"There was absolutely no basis to conclude that Oscar has any advantage at all over able-bodied runners."

Not only does this statement disagree with IAAF test results on the blades (where 30% advantages were found for some variables), but based on Herr's testimony and the evidence gathered, Pistorius is appealing to legalize the use of the carbon-fibre Cheetahs for competition. The IAAF, the governing body for the sport, are adamant that the technology (not the person, take note) should be banned, because it introduces engineering innovation and future technology that affects the credibility of the sport. It is therefore a statement to take note of. So who is Professor Hugh Herr?

Well, consider that two weeks ago, the self-same Professor Hugh Herr was quoted in an excellent article by ESPN which said the following:

A bilateral amputee professor named Hugh Herr works here (at MIT). If anyone can predict what sports will look like in 2050, it's Herr, who lost his legs 26 years ago in a climbing accident. Herr wears robotic limbs with motorized ankles and insists he doesn't want his human legs back because soon they'll be archaic. "People have always thought the human body is the ideal," he says. "It's not."

Next, consider the fact that when he's not delivering conflicting scientific testimonies, Hugh Herr is a co-developer of Ossur prosthetic limbs, and a collaborator, contracted expert and speaker for Ossur at their various functions. Ossur, incidentally, are the company who make the high-tech carbon fibre blades Pistorius uses to run - they stand to make millions in commercial exposure if he can run in Beijing, and have even previously pressured the IAAF to allow him to participate.

These are the same blades which were independently tested by the IAAF and found to provide advantages in the range of 20 to 30% compared to normal limbs. This testing proved what had been suggested for months, based on physiological theories for the case. There was little new information, just proof of known physiology and an enormous advantage. Remarkably, the research later carried out by Pistorius (which has yet to be published or peer-reviewed, and which was done in "top-secret" with not a single independent witness or representative from the IAAF present) found the opposite--no advantage, and 20-30% was made to disappear.

So the million dollar question, with perhaps a (multi) million dollar answer, is how much does it cost to buy a scientific opinion?

The CAS process: Legal system meets science

The verdict from the hearing is expected relatively soon - the CAS usually takes much longer to get verdicts out, but given the proximity to the Beijing Olympics, it was considered important to deliver some verdict sooner, which is commendable. Once that verdict is announced, we'll certainly cover the outcome and evaluate the "science" if it is ever presented, though we're not optimistic of the evidence ever coming to light.

But for now, the last week has produced some interesting emails and observations that examine the CAS process a bit more.

First, the CAS process is an interesting one, and in particular, this kind of case is one that I am reliably informed is particularly "unpopular" within the legal boundaries of arbitration. Because effectively, what the CAS does is put two groups of disputing scientists in front of a panel of three non-scientists who have to make a judgement call, on an argument of science. Under these circumstances, with "duelling experts", the arbitrators are not equipped to evaluate the scientific facts and thus more likely to adopt an attitude saying that "unless one side can prove beyond doubt that there is an advantage, we must allow the technology into the sport".

The hearing therefore does not end up evaluating the "scientific debate," and therefore it becomes rather "advocacy by science." The scientific experts become not just presenters of the science, but advocates, and the credibility of the scientist becomes more meaningful than the content presented by them. This is however true for any "expert witness", and this case is no different. It's an interesting dilemma for the CAS arbitrators, however, and one wonders whether or not they have the option to consult with an independent scientist to sort through arguments that are well outside their field of expertise?

The Scientific expert: How much does an opinion cost? Science for sale

But more than this, the bigger issue here is incentives and scientific credibility. This past week, a group of post-graduate university students with whom I work with was tasked with looking scientifically at the marketing claims made by a group of supplement companies. I'll spare the details, though perhaps supplements is a topic we'll look at in the future, but the single "take home" message I got out of the scientific analysis, is that "Science sells, if you're prepared to sell yourself for money as a scientist."

Therefore, whenever you are evaluating a scientific argument of any description, you have to ask two key questions:

1. What is the incentive? WHY are they saying this? In the case of supplement companies,they make claims because they want you to hand over $50 to buy their product. But why would a scientist endorse a product and declare it to be "proven"? The supplement industry has thrived thanks to sometimes tenuous associations with scientists and universities, but often, the scientist endorses a product. Why? See question two.
2. How does the money flow? Who is ultimately splitting that $50 (which rapidly grows into billions of dollars)? Because as soon as some of that money makes its way into the scientist's pocket, then you have the likely answer to question #1. Because suddenly, the scientist is revealed to have a perverse incentive to change his "scientific" opinion and sell out to the lure of commercialism! That is not to say that scientist cannot do this - by all means, if one's expertise can earn a salary, go for it. But scientifically, that scientist's "objective" opinion is undermined, because he can longer adopt a neutral or contrary position without fear of losing money or financial support. If money is paid by Company X, then the scientist must support Company X, because failure to do so, or supporting Company Y, threatens his meal ticket and lifestyle. Therefore what we have is a classical conflict of interest situation.
   

The Pistorius case and the quotes at the top of this post are a great example of incentive conflicts in science. I've no doubt that Professor Herr is a great scientist. That's not in question, and our aim here is not to interrogate Prof Herr's scientific career. But it strikes me as perverse that the same scientist can, in one situation, say that "People have always thought the human body is the ideal. It's not".

And then, in a later situation, the same person can make the statements to the effect that the same technology which he believes is better than the human leg should be allowed into the sport. Bear in mind that the entire debate around the high-tech carbon fibre Cheetahs is not whether Oscar Pistorius should run (though this is how it has been portrayed). Instead, it is whether the TECHNOLOGY he uses should be legal, and this has future implications for the sport. Pistorius simply is the context or vehicle around which this debate is centered.

This expert clearly believes the prosthetic technology will soon become superior (if it isn't already - he doesn't want his own soon to be "archaic" limbs back, you will recall), yet has no qualms about advocating for its entry into the sport with his expert opinion. So what is the difference between these situations? Could it be commercial?

Of course, those who have much to gain financially from Pistorius' participation in the Games (and there are many, including Pistorius himself, his family, sponsors etc.) will argue this point, but at the very least, these incentives, as well as the flow of money, should be disclosed, for the sake of transparency. It is good practice and not uncommon for scientists to make a full disclosure prior to presenting or publishing their work. In fact it is a standard part of the process for many scientific journals.

Speaking of transparency, the next problem is the circumstances around which the testing was done. When the IAAF tested Pistorius in October, he was given opportunities to invite a "support team," which could include his own scientist(s) if he wished. He was given the chance to contribute to the testing process, even so far as to develop the outcomes which would be tested. That testing was completely independent, verified and agreed upon by both parties before it was ever carried out.

However, in Pistorius' own "top-secret" testing, not a single IAAF or independent witness was present or invited. Instead, a group of scientists who have a year-round, close relationship with Ossur, who stand to make millions in commercial exposure if Pistorius can run, completed the testing in private. To date, that testing has not been released, and I don't expect it will be.

The reality is that if any scientist was asked (and was willing), they could sit in their office and make up results, out of their heads, that could clear Pistorius of any advantage! Fictional, imaginative science could develop a series of graphs and tables to confirm the absence of any advantage in most scenarios, including this one. Let it be known by all that this is not science. Science is peer-reviewed, subject to scrutiny, evaluation and challenge. So where is the scientific process here? I am astonished that this "scientific evidence" was even admitted to court, but then the legal system never fails to throw up surprises - freedom for sale, anyone?

Wrap up and look ahead

Time will tell how the CAS rule on this issue. If, as is entirely possible, the three judges are not able to weed out the "science" from the science, and Pistorius is cleared to run, I will propose what I'll call "Operation West African Gold", where any willing sponsor (Nike, perhaps, who would not baulk at this opportunity based on their aggressive role in the present case) goes into West Africa, home of the fastest sprinters in the world, and finds 30 children, aged 10 to 14, with bilateral amputations. By 2020, they'll be winning half the golds in the Olympic Games, and the 400m world record will be 40 seconds. This is the Pandora's box which "science advocates" would open should the CAS rule in favour of Pistorius.

Ross