Dr. Weeks’ Comment: More than two decades ago, I subscribed to The Journal of Irreproducible Results which, by publishing outlandish (and irreproducible) papers in a scientific manner, targeted “hypocrisy, arrogance, and ostentatious sesquipedalian circumlocution”. My favorite JIR paper described the “anti-depressant” effect noted by staff at an inpatient gero-psychiatriy ward at Dartmouth when an elderly, senile gentleman with a libido and playful spirit both exceeding his capacity for judgment, began “goosing” elderly women on the ward. His groping the ladies had the quite remarkable effect of raising the mood and reducing “clinical depression” on the ward. This, of course, constituted an irreproducible result which nonetheless was worth of publication in the JIR.
Now we learn that most publications in uber-respected peer-reviewed scientific journals are also…. get this…..irreproducible! You hear me correctly. Irreproducible! Why is this shocking? Because the sine qua non of science is that it embraces a methodology which allows for confirmation and reproducible findings. If the findings are not reproducible – the very same findings which dictate how billions of dollars are spent in the name of health care – then logic suggests some degree of both the following to be true: 1) the paper which was published was based upon fraudulent science or 2) the methodology was sloppy. In either case, this is an all-hands-on-deck”, 5- bell alarm which the NIH and all government funding sources must attend to. That is, if they are committed to serving science and not simply marketing agents for Big Pharma.
Here is a summary of the problem:
“…The recent questions raised about the reproducibility of biological and medical research findings is forcing scientists to embark on a soul-searching mission. It is likely that this journey will shake up many long-held beliefs. But this reappraisal will ultimately lead to a more rigorous and reliable science..”
Here is the paper which describes the problem.
(Thanks to Dimitri for sharing this with me!)
SUNDAY, SEP 1, 2013 07:30 AM PDT
Cancer research in crisis: Are the drugs we count on based on bad science?
A paper finding that important experiments can’t be reproduced suggests new drugs could be based on bad science
The cancer researchers Glenn Begley and Lee Ellis made a rather remarkable claim last year. In a commentary that analyzed the dearth of efficacious novel cancer therapies, they revealed that scientists at the biotechnology company Amgen were unable to replicate the vast majority of published pre-clinical research studies. Only 6 out of 53 landmark cancer studies could be replicated, a dismal success rate of 11%! The Amgen researchers had deliberately chosen highly innovative cancer research papers, hoping that these would form the scientific basis for future cancer therapies that they could develop. It should not come as a surprise that progress in developing new cancer treatments is so sluggish. New clinical treatments are often based on innovative scientific concepts derived from pre-clinical laboratory research. However, if the pre-clinical scientific experiments cannot be replicated, it would be folly to expect that clinical treatments based on these questionable scientific concepts would succeed.
Reproducibility of research findings is the cornerstone of science. Peer-reviewed scientific journals generally require that scientists conduct multiple repeat experiments and report the variability of their findings before publishing them. However, it is not uncommon for researchers to successfully repeat experiments and publish a paper, only to learn that colleagues at other institutions can’t replicate the findings. This does not necessarily indicate foul play. The reasons for the lack of reproducibility include intentional fraud and misconduct, yes, but more often it’s negligence, inadvertent errors, imperfectly designed experiments and the subliminal biases of the researchers or other uncontrollable variables.
Clinical studies, of new drugs, for example, are often plagued by the biological variability found in study participants. A group of patients in a trial may exhibit different responses to a new medication compared to patients enrolled in similar trials at different locations. In addition to genetic differences between patient populations, factors like differences in socioeconomic status, diet, access to healthcare, criteria used by referring physicians, standards of data analysis by researchers or the subjective nature of certain clinical outcomes – as well as many other uncharted variables – might all contribute to different results…..