Peer-review: “accurate measures of the prevailing bias”

Dr. Weeks’ Comment:   The standard of care is a nebulous, transitory consensus which, at best, is an intelligent distillation of relevant medical knowledge offered as guidance for the practicing doctor who, in the trenches and not the ivory tower of abstract academia,  must apply myriad data to the “N of 1″ which is the patient before him or her. How is the standard of care arrived at?  Primarily through consideration of published peer-reviewed” articles by “alpha” thought leaders who serve on committees of medical associations which offer practice guidelines. As of yet, the degree of humility of these committees is such that the term “guideline” remains in vogue,  as opposed to what may morph in the near future into the more constraining term: “edict”.   Therefore, presently, the responsibility to determine appropriate treatment resorts to the practicing doctor, in collaboration with what is, ideally, an informed patient who, after full consideration, freely gives consent to treatment. Theoretically, this system offers the optimal arrangement: rigorous, consensual “group think”  blended thoughtfully with  “beyond the standard of care” individual innovations.  But,  if we peek behind the curtain of peer-reviewed process, things get considerably more murky and my contention is that more responsibility should remain with the patient (an increasingly informed internet savvy consumer reinforced by the integrity of the free market) as well as  the medical doctor (who must integrate best practices as well as market forces to deliver a product (“health care”) which is valued enough by paying customers (informed consumers/ grateful patients) for the doctor to remain in business.

In contrast, today’s health care discounts both consumer choice and market forces being delivered, as it is,  in a vacuum where outcomes are less important than methodology.    (You read that correctly: if a patient dies after receiving the standard of care: no fault.  If the same patient thrives after receiving non-standard of care: fault!  In particular, medical board criticism for the doctor having subjected the patient to “unreasonable risk of harm” simply because non-standard of care treatments were offered. The medical board has no appreciation of the successful outcome. Only in the practice of medicine does the process trump the result. Think about it: “The surgery was a success but the patient died”.)   Practically speaking, the effect of third-party payers is to obfuscate the actual costs of services by shifting direct accountability away from the customer (who is most qualified to determine whether the value received is commensurate with the fees charged) and onto a vast insurance machination whereby charges for CT scans, brief bed-side consultation during hospital rounds, chemotherapy drugs and even hand-held urine collector can be charged at an exorbitant and unsustainable rate of 400 to 4000 fold mark-up!

Back to how the standard of care is established and maintained.  Just how accurate and reliable are peer-reviewed articles as a beacon for steering us towards a meaningful standard of care?  Read John Ioannidis’s excellent article here and abstracted below.  

“…claimed research findings may often be simply accurate measures of the prevailing bias…”  


” There is increasing concern that in modern research,  false findings may be the majority or even the vast majority of published research claims. However, this should not be surprising. It can be proven that most claimed research findings are false…”


Just as the peer-review process  begs for a good-faith overhaul, so too does the current health care system beg for more accountability, more efficiencies and more empowerment of the patient who ought to be free to select treatments above and beyond the current standard of care. 


Why Most Published Research Findings Are False

  • John P. A. Ioannidis


There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.

Read the entire article here. 


Many thanks to Dr. Ivan Oransky, MD
Executive Editor, Reuters Health
Co-Founder, Retraction Watch
Founder, Embargo Watch
Adjunct Associate Professor, New York University’s Science, Health, and Environmental Reporting Program
Treasurer, Association of Health Care Journalists
Clinical Assistant Professor of Medicine, New York University School of Medicine


Leave a Comment

Your email address will not be published. Required fields are marked *