Discussants: Justin Geller, Jonathan Taylor, and Jordan Weil
Synopsis of David Freeman's Lies, Damned Lies, and Medical Science
Something is rotten in biomedical research, says Dr. John Ioannidis. And, if you're ever going to make a serious decision based on scientific findings, you should consider this assertion: financial and psychological incentives--along with academic politics--drive researchers to publish biased papers. Short of opting for a career in research over the clinic, what's an aspiring dentist/doctor/nurse/vet to do?
You may have already condemned the drug companies at this point. They certainly are part of the problem. The history of "selective reporting" of trial results is well established (see, e.g., Chan et al., 2004). In response, many journals now refuse submissions from studies not pre-registered with the editor. Though a valuable mechanism, bias has not been eliminated. There is also question bias: if the comparison therapy is a drug already known to be inferior to others on the market, a new therapy has very good chances of showing significant benefit. "Of course!", you say, "you can get published for a finding like that?" Big Pharma is an easy foil.
Further validating established frameworks rarely brings great acclaim, so there is significant pressure to interpret results to the investigator's favor if not building in bias to the methodology. Career advancement is trumping good science. Improbable results are sexier than plausible ones. But villainy doesn't require villains. Though there are certainly instances of willful malfeasance, the biases that crop up may not be by conscious design. Research bias may instead come from unconscious urges to produce a significant finding. There's no market for null results. It is difficult to secure funding for result reproduction, and can be professionally awkward to refute a colleague's results. While a researcher may not have a pecuniary interest in a company hoping to bring a product to market, they have an indirect yet effective financial interest in advancing their career.
It is important to note that the bad-paper problem is not limited to little-regarded niche journals desperate to fill out an issue. In a meta-analysis of the most highly regarded research findings (as measured by stature of the journal and number of citations), Dr. Ioannidis's research team found a troubling trend even in those papers that made intervention effectiveness claims. Of the ~3/4 that were subjected to verification, 41% were found to be either flatly wrong or significantly exaggerated in their claims. These continued to be cited for years after their refutations, some more than a decade after they were disproved.
There's one topic that I'd like to make a regular feature of these discussions:
How does this affect us as prospective health professionals?
Admittedly, this is pre-MD/DO focused: how should we incorporate the Dr. Ioannidis's findings into our learning and practice? Faulty research results are not going to disappear from the literature. How do medical schools prepare us interpret professional research findings? They will train us to be physicians, but how effectively do researchers and clinicians communicate? Will we be taught how to read professional research output?
I don't believe that reading a research paper is not a matter of applying common sense: professional research have their own shorthand. To echo a point that Dr. Ioannidis made in the interview, researchers and clinicians focus on different things and use their own vocabularies. Papers' authors assume a base of knowledge that a clinician learning simply may not have. Once we're practicing clinicians (assuming that's where most of us are going), we're not going to have the time to keep up on entire fields. My impression is that clinicians do keep up on scientific findings (and this is in NO WAY a knock on clinicians: researchers in disparate fields can't read each others' papers effectively either) but they simply don't have the time or training to comb through all of the minutiae the latest findings.
ReplyDeleteNot to sound like a broken record, but how do we deal with this as clinicians? That's not a rhetorical question. Please suggest something.
There are some compounding factors in that researchers and doctors (and anybody who has an expertise on much of anything) is going to be hesitant to admit to themselves that they may have been taken in. The human ability to resist being labeled a sucker has been a prime motivating force throughout human history. Maybe hubris is the wrong word for it, but pride comes before the fall, or something like that.
Justin and Johnny made a few interesting points (guys, please feel free to flesh these out or correct me in your own posts):
ReplyDelete- Justin made a comment about how this type of meta-research finding will affect the legal standard of the objective vs. subjective test in determining liability during malpractice suits. Justin, could you take a few minutes to expand on that?
- Johnny made a point (which I rephrased because I'm kinda imperious like that) about a need for a culture that accepts mistakes in science and the need for greater focus on publishing null results.
- Going back to the "how will this affect our practice" motif, Johnny raised an excellent question of how a practicing physician should deal with an engaged patient bringing in professional research papers that relate to their illness and questioning their doc about how they should integrate the latest findings into their care.
- Justin wonders how much of a difference there is between the US model of funding and the European, and how much that affects the incentives for bias. Does American entrepreneurialism stack the deck against us, or does it not make too much of a difference? Are US researchers cutting corners for lack of funding? Is there just too little money for verification/refutation work?
The article briefly mentions a patient that Tatsioni will later examine. When talking about that patient Tatsioni mentions that just taking the time to do a proper bedside chat and knowing her patient's history will probably give her a better idea of what is going on than ordering a whole slew of tests and prescription medications.
ReplyDeleteSo, I'm thinking that part of the problem is that there is a need for practicing clinicians to get to know their patients better, rather that treating them as an "errant number" to get back into line. This kind of highlights the need for primary care physicians that can really get to know their patients well, but can also be applied to any other field of medicine. Of course, these days PCP's are under a lot of pressure to do the exact opposite - order lots of tests, spend less time with each patient... so there's that.