BMJ analysis reveals widespread publication/selection bias in research
Filed under: Conflicts of interest, Government, Health data
Reporting on a study released by BMJ and characterized as an almost existential threat to the medical research system by Dr. Harlan Krumholz, the Milwaukee Journal Sentinel’s John Fauber writes that “Drug research, even from clinical trials sponsored by the federal government, routinely is suppressed, harming patients and increasing health care costs.” The emphasis is mine, the strong language Fauber’s.
The conclusions are based on a survey of meta-analyses of individual participant data, which the authors broke down by data source characteristics and publication status. The work is heavy on statistical analysis, but even lay readers can understand the broad strokes of what appears to be a widespread issue.
Steve Nissen, the lead author of the analysis, said 35 of the 42 studies he looked at were unpublished and were obtained only because a court case required the drug’s maker, GlaxoSmithKline, to turn over the data.
And it isn’t just pharmaceutical companies’ financial concerns driving the suppression, Nissen and his coauthors found. At that point, it may more of an issue of confirmation bias and other problems which have always lurked within academic research.
A surprising finding in the BMJ analysis was that serious lapses occurred even in clinical trials funded by the National Institutes of Health.
That research showed that less than half of NIH-funded clinical trials were published in a medical journal within 30 months of the completion of the trial and after 51 months, one-third of trials remained unpublished.
Online comments lead to BMJ’s disclosure of ‘competing interests’
Filed under: Conflicts of interest, Health journalism
In BMJ, Bob Roehr wrote about a report published by German researchers in the Canadian Medical Association Journal describing an apparent tendency for journals that accept pharmaceutical advertising to publish more positive drug-related articles than those that depend on subscription dollars to pay the bills. The study and the Roehr’s summary are good reading in their own right, but the comment section is where things really get interesting.
There, Age of Autism UK editor John Stone points to a commentary penned by the Alliance for Human Research Protection’s Vera Hassner Sharav and draws into question BMJ’s sources of funding. His main focus is the tension between that publication’s Andrew Wakefield investigations and its receipt of money from an arm of Merck.
Sharav’s language is somewhat incendiary, but it’s BMJ editor Fiona Godlee’s response to her commentary (and Stone’s post) that push the whole thing into the realm of the remarkable. Godlee weighs in on everything right there in the comment thread, admitting that BMJ had not disclosed those conflicts of interest in the Wakefield stories simply “because it didn’t occur to us to do so,” given that it was a story focused on research fraud rather than upon vaccines and medicine.
Although Vera’s claims may seem far fetched on this occasion, she is right that we should have declared the BMJ Group’s income from Merck as a competing interest to the editorial (and the two editor’s choice articles) that accompanied Brian Deer’s series on the Secrets of the MMR scare. We should also, as you say, have declared the group’s income from GSK as a competing interest in relation to these articles. We will publish clarifications.
The whole chain of events is a promising sign that increased interactivity in online publications may lead to increased transparency, and it’s well worth reading, at the very least, all of Roehr’s story and the comments that follow it. All the key bits are there.
Adding context to embargo-driven journalism
Over at the Nieman Journalism Lab, Matthew Battles latches onto John Rennie’s column about the future of science journalism, then talks to Ed Yong and AHCJ’s own treasurer, Ivan Oransky, M.D., about embargoes, timelines and cutting through the noise.
Battles focuses on the effort to pursue context in a profession that is so often at the mercy of academic journals. After all, as he writes in his introduction, “The events that science journalists publish about most frequently are themselves acts of publishing: the appearance of research papers in peer-reviewed journals.” The rest of his piece will serve as a handy primer for anyone looking to understand why that particular state of affairs is so pervasive and persistent.
For AHCJ members who want to know more about responsibly covering studies and how to recognize and report the problems, limitations and backstory of a study, as well as publication biases in medical journals, be sure to see “Covering Medical Research.”
Study authors don’t always have access to raw data
Filed under: Conflicts of interest, Hot Health Headline, Pharmaceuticals, Studies
John Fauber of the Milwaukee Journal Sentinel reports on outside authors of drug company-funded studies who do not have access to the raw data behind the study.
In the latest installment of Side Effects, a series examining doctors, drug companies and conflicts of interest, Fauber reports that a researcher who co-authored a study of Multaq that led to the drug’s FDA approval “vouched for the accuracy and completeness of the study despite not seeing the raw data.”
The researcher, Richard Page, chairman of the department of medicine at the University of Wisconsin School of Medicine and Public Health, says he trusts the Sanofi-Aventis, the drug’s manufacturer.
Fauber delves into the differing editorial policies of the Journal of the American Medical Association and the New England Journal of Medicine when it comes to studies in which drug company employees perform the data analysis.
In the case of the Multaq trial, published by the New England Journal of Medicine, the FDA got unanalyzed raw data and did its own analysis. Fauber reports that “one FDA panel member questioned differences between the information included in the published February 2009 study and what was submitted to the FDA.”
Related
- Uncovering conflicts of interest in medicine, research
- Health Journalism 2010: Journalists encounter obstacles in identifying conflicts of interest in medical research
‘Salami slicing’ yields many articles from one trial
Filed under: Health journalism, Hot Health Headline
As discussed by Toronto Star’s Stuart Laidlaw and Pharmalot’s Ed Silverman, the journal Psychotherapy and Psychosomatics recently drew attention to the practice of publishing multiple journal articles from the results of one clinical trial in a study they titled “A Case Study of Salami Slicing: Pooled Analyses of Duloxetine for Depression” (PDF). Duloxetine, for the record, is an Eli Lilly drug better known as Cymbalta.
Salami slicing is well known (and disliked) by top-flight journal editors who, Laidlaw said, often require authors to “declare if any part of the underlying research has been – or soon will be – published elsewhere.” Still, the researchers behind the Psychotherapy and Psychosomatics paper found it remains widespread, at least in the case of Cymbalta.
The authors speculate that the practice is propelled by two key factors: researchers who want to increase their exposure and publication counts and pharmaceutical manufacturers who want to “widely disseminate” the results of positive studies and create an artificially inflated base of scholarly support for their products.
Researchers defend the practice as cost-effective, saying that medical trials can be prohibitively expensive, and salami slicing is one way to extract as much research as possible from one trial. Additionally, they say that trials are intentionally designed to be sliced, and that there are often multiple separate issues that each warrant their own manuscript.
Laidlaw also found at least one researcher who took issue with the semantics of the practice’s sensation-friendly label.
“I would be careful of anything that’s given a folksy name,” says Dr. Ralph Meyer, director of the National Cancer Institute of Canada’s Clinical Trials Group at Queen’s University.
Oransky: Medical study embargoes serve whom?
Embargoes, a fairly frequent topic of discussion on Covering Health, seem to bring out strong feelings in some people.
In one recent example allegedly involving embargoes, TheStreet.com’s Adam Feuerstein attempted to combat a rumor that the New England Journal of Medicine would be publishing an article about the experimental lung cancer drug seliciclib - a rumor that was seemingly driving up the stock price of Cyclacel Pharmaceuticals.
Feuerstein looked at his advance copy of NEJM and reported that no such article was coming out. He was immediately accused by a commenter on the story of having broken the NEJM’s embargo.
Photo by Billingham via Flickr
As Reuters Health Executive Editor - and AHCJ board member - Ivan Oransky asked, “Is saying what’s NOT in an embargoed journal breaking the embargo?” Oransky concluded that it was not a case of breaking an embargo. The ensuing discussion on Twitter and in the comments on Feuerstein’s article is interesting. For the record, the NEJM apparently sided with Feuerstein, according to a commenter on the story.
Today, Oransky weighs in on the wider topic of who is served by embargoes on medical studies:
Two weekends ago, at ScienceOnline2010, I heard an interesting thing about embargoes. Connie St. Louis, who directs the science journalism masters’ program at City University, London, told an audience that one of the reasons for embargoes on scientific journal studies is that with more eyeballs on the study before publication, it’s more likely researchers will catch flaws in papers, which can then be pulled.
In other words, just as the FDA requires drug companies to monitor drug side effects in large populations once a drug is approved, in what’s called post-marketing surveillance or phase 4 testing, because such side effects may not show up in relatively small trials, this is a sort of post-acceptance peer review. So if a reviewer doesn’t catch an error during the normal course of peer review, journals can use the embargo period as a backstop.
I hadn’t heard that before, and I consider myself fairly well-acquainted with the arguments for and against embargoes. But it reminded me again that for all the talk of embargoes serving the public by allowing reporters to write more-informed stories, there are serious questions about whether journals are the group that gains most from embargoes.
As it turned out, I was in the midst of another episode that reminded me of that. Last week, the Cochrane Library published their quarterly set of reviews. Among them was a review of whether opioid drugs, when used as prescribed, carried a high rate of addiction. Cochrane researchers found that they don’t. That’s not the biggest research finding ever, but when you run a health news wire service filled with dozens of stories a day, like I do, it’s something worth covering. Plus, opioid dependence remains a big issue, for celebrities and lawmakers alike.
That study was embargoed for 7 p.m. Eastern on the 19th. Around the same time I was reviewing it, the Annals of Internal Medicine press packet hit my desk. That packet — embargoed until 5 p.m. Eastern Monday the 18th — included a paper that said high doses of opioids, even if prescribed, increased the risk of overdose. Again, not the most shocking study ever, but researchers and advocates continue to debate whether these drugs, when used in prescribed doses, are dangerous.
I figured the best way to serve our readers would be a story that included both of these studies, both as context for the other. Trouble was, if I ran one based on the Annals study, I couldn’t mention the Cochrane review. And if I waited for the Cochrane review’s embargo to lift, a competitor might run the other story. (Yes, we think about these things.)
So I emailed Jennifer Beal, who runs media relations for the Cochrane Library’s publisher, Wiley, explaining the situation, and asking her if she would consider moving the embargo. She returned my message right away, saying politely that she couldn’t, and explaining why, in a message that I found thoughtful.
“We thought about it very carefully but felt that our guiding principle with embargoes is that we are giving media an opportunity to investigate a story fully without the pressure to publish immediately, so that the story is still ‘new’ on the day the research gets published, and is therefore available for public consumption,” she wrote. “If we were to agree to a moved embargo, it would mean that you…would be writing about a story where the research was not available for the public to read if they wish and make up their own minds.”
The opioid study, she noted, was “one of approximately 160 articles publishing on Wednesday; this is a big operation where the publication schedule is planned out a long way in advance, so it is not possible to move forward the publication date.” (In a long-planned move, Cochrane is now going monthly, which will distribute the reviews more evenly.)
Based on resource constraints, I decided we could only run one story on opioid addiction or overdose last week. We ended up running a story on the Annals study, which seemed a bit more newsworthy than the Cochrane review. As it turns out, a number of news organizations covered the Annals study — Seattle’s LocalHealthGuide ran an item, which the Seattle Times picked up; The Seattle Post-Intelligencer, and our competitors HealthDay and Bloomberg, among others. As far as I can tell, only a site called MedIndia ran a story by the Health Behavior News Service pegged to the Cochrane review.*
(I should say here that I’m a big fan of the Cochrane Library, access to which is an AHCJ benefit. Some have criticized them for rejecting everything other than randomized double-blinded controlled clinical trials, but I’d rather see more rigor than less, given how often hype and poor evidence tend to rule the day. So this isn’t really about a problem at Cochrane. It’s about how embargo policies, whether well-intentioned or not, often make me wonder whether they serve the public.)
So whom did this Cochrane embargo serve? I’d argue it didn’t serve the public, because we and others couldn’t include news of it in the story we did decide to run. You might even say it didn’t serve Cochrane either, since I’m guessing many of my colleagues decided not to run something on it for similar reasons. And their rationale for not moving the embargo at least had the public in mind.
The episode reminded me of the CDC-autism embargo fiasco last fall, in which the CDC and the journal Pediatrics refused to lift an embargo on autism rate data that many in the autism advocacy community had already reported on. To me, the Cochrane decision was more justifiable, even if I didn’t like it.
Many have questioned embargoes, notably Vincent Kiernan, in his 2006 book “Embargoed Science.” Kiernan makes a convincing argument that embargoes serve journals most, by giving reporters something to cover every month or every week. Nowadays, that’s even more true, I think, as I see an increase in papers embargoed just 1-2 days, rather than the typical 5-6.
Still, when you run a high-volume news service, as much as you’d like to, it’s not reasonable to reject all embargoes in favor of 100% enterprising reporting. Our clients would be very unhappy, and justifiably so. Instead, we can try to cover studies with as much skepticism and context as possible. But when I hear yet another reason why embargoes might help journals, as I did at ScienceOnline2010, I’m only encouraged more to challenge the idea that embargoes are there for the public, and at least force journals to defend how they handle them.
Ivan Oransky, M.D., is executive editor of Reuters Health and AHCJ treasurer. Follow him on Twitter, where he periodically gets into debates about embargoes, among other things.
*Update:
After this post was published, we heard from Lisa Esposito, editor of the Health Behavior News Service, who tells us that Medscape and Elsevier Global Medical News did cover the Cochrane opioid review.
41 additional journals make the Medline cut
Looking for the latest edition of Advanced Emergency Nursing Journal, the Journal of Patient Safety or the Saudi Journal of Gastroenterology, the official journal of the Saudi Gastroenterology Association? You’re in luck! They are among 41 publications to recently be accepted by the National Library of Medicine for inclusion in the Medline database.
Three times a year, the National Library of Medicine’s Literature Selection Technical Review Committee meets to review Medline’s roster of about 5,000 journals. At each meeting, the committee considers about 140 new journals, evaluating each based on their scientific merit and upon where they fit in the NLM’s ideal literature mix. To be included, journals need to produce high-quality, medical-related content with high editorial and production standards. It also helps if the abstracts, at least, are in English.
Related
AHCJ Tip sheet: Mining NLM databases: PubMed, Medline and more
Medscape drops publication of articles
The Medscape Journal of Medicine announced Friday that it is ceasing publication of new articles.
The Journal was one element of the Medscape Web site, which will continue “focusing on our role as an aggregator and interpreter of medical information and not as the primary source for original scientific articles.” The announcement says “all previously published articles will remain published and available on the Medscape platform, indexed in MEDLINE/PubMed, and – for the next several years – in full text at http://www.pubmedcentral.nih.gov/ as well.”
Ivan Oransky, managing editor for online at Scientific American and an AHCJ board member, writes about the changes and about open access medical journals in general.
-
Join us in Atlanta
for Health Journalism 2012
Fellowship applications
due Feb. 27
New at HealthJournalism.org
- Data gives depth to investigation of patient safety
- What reporters should know about Alzheimer’s and related dementias
- Apply for fellowships to attend Health Journalism 2012
- AHCJ asks Supreme Court to permit broadcast of arguments in health reform case
- Tips for print, online journalists to embrace broadcast opportunities
- Numbers reveal how often, or how rarely, states check doctors’ disciplinary records
- How I did it: Falsified charts in nursing homes linked to patient care
- Access Denied: Looking at a lack of health care availability
- Transparency & medical errors: Public reporting of patient harm is a potent force
