Society ‘snookered’ by research that isn’t new
Peggy Peck of MedPage Today found that research presented as new at the European Society of Cardiology’s annual meeting this weekend was actually published in July, despite the society’s requirement that information submitted for presentation must be new, unpublished data.
When asked by MedPage Today to point out the “news” in the Hot Line presentation, STAR lead investigator Bodo-Eckehard Strauer, MD, of the Heinrich Heine University of Düsseldorf, Germany, said the news was that bone marrow cell therapy significantly improved survival in patients with chronic cardiomyopathy, which he illustrated with a slide showing a Kaplan-Meier curve – the same graph that was published in the July issue of the European Journal of Heart Failure. Moreover, every data slide in Strauer’s presentation matched the tables in the published paper.
Following questions from MedPage Today, the organization acknowledged its error and has announced the researcher will not be allowed to present at its meetings for two years. Roberto Ferrari, M.D., president of the society, said the research had been accepted for presentation because they thought it had new data but that “We were snookered.”
Hearing on public access to research will be online
Filed under: Health data, Health journalism, Public records, Studies
Representatives of a number of medical- and publishing-related organizations will testify today at a hearing on “Public access to federally funded research” before the Information Policy, Census and National Archives Subcommittee of the U.S. House of Representatives’ Committee on Oversight and Government Reform.
You can catch the webcast of the hearing at 2 p.m. EDT. Update: It appears the hearing is not being webcast.
(Hat tip to the Electronic Frontier Foundation for the webcast link.)
Earlier
- Bill would require public access to research
- AHCJ: Proposal would be blow to public access
- Progress on open access issue not what it seems
Study authors don’t always have access to raw data
Filed under: Conflicts of interest, Hot Health Headline, Pharmaceuticals, Studies
John Fauber of the Milwaukee Journal Sentinel reports on outside authors of drug company-funded studies who do not have access to the raw data behind the study.
In the latest installment of Side Effects, a series examining doctors, drug companies and conflicts of interest, Fauber reports that a researcher who co-authored a study of Multaq that led to the drug’s FDA approval “vouched for the accuracy and completeness of the study despite not seeing the raw data.”
The researcher, Richard Page, chairman of the department of medicine at the University of Wisconsin School of Medicine and Public Health, says he trusts the Sanofi-Aventis, the drug’s manufacturer.
Fauber delves into the differing editorial policies of the Journal of the American Medical Association and the New England Journal of Medicine when it comes to studies in which drug company employees perform the data analysis.
In the case of the Multaq trial, published by the New England Journal of Medicine, the FDA got unanalyzed raw data and did its own analysis. Fauber reports that “one FDA panel member questioned differences between the information included in the published February 2009 study and what was submitted to the FDA.”
Related
- Uncovering conflicts of interest in medicine, research
- Health Journalism 2010: Journalists encounter obstacles in identifying conflicts of interest in medical research
Coverage of bacon, cell phones doesn’t add up
Zoe Williams, a columnist for the UK’s Guardian, weighs in on coverage of two recent studies, saying that the Daily Mail “takes the role of the Friend Who Exaggerates.”
Williams takes issue with how the paper reported a study about cellular phones and the risk of brain cancer and how it reported on a Harvard study about bacon and heart disease.
She points out that, in the cell phone study, “there were 10 usage groups, ranging from very low to very high. In the very highest group – those reporting using their phone for 12 or more hours a day – there was a raised chance of both glioma and meningioma.” Williams and the study’s author agree that level of use is improbable.
While other papers reported that the study did not find a statistically significant increase in risk, The Daily Mail ran the story with a headline that says “Long conversations on mobile phones can increase risk of cancer, suggests 10-year study.” She also notes that a dose response is missing and that there is no evidence that radio waves – emitted by cell phones – cause cancer.
In the story about bacon and heart disease – which runs with a headline that declares “A bacon sandwich a day raises risk of heart disease by half” – she notes the Daily Mail reports “the risk of heart disease goes up by 42% with every two-ounce (about 56g) serving of processed meat.” Williams calls the 42 percent increase as small in epidemiological terms. The Daily Mail also doesn’t tell readers if that is the absolute risk or the relative risk.
Williams, who calls such stories “thrill-tainment,” concludes that “Health journalism (and it’s not just the Mail) needs more scientific credibility, even to function as entertainment.”
Report measures health factors at county level
Filed under: Health data, Health policy, Public health, Studies, Tools
A county-by-county collection of reports set to be released tomorrow could be a good source for local data on a number of health factors. The County Health Rankings, a collaboration between the Robert Wood Johnson Foundation and the University of Wisconsin Population Health Institute, is the first of what is expected to be an annual look at health within each state.
The rankings will “show how counties measure up within each state in terms of how healthy people are, how long they live, and how important factors affect their health, such as tobacco use, obesity, access to healthcare, education, community safety, and air quality,” according to a press release.
A Feb. 17 briefing will include experts representing public health, health policy, education, and business who are expected to discuss the rankings and ways that communities can become healthier. The briefing, which starts at 9:30 a.m. EST, will be webcast. You can RSVP for this event online and a link will be sent to those who RSVP as soon as it’s available. For those in Washington, D.C, the briefing will be at the Columbus Club at Union Station, 50 Massachusetts Ave. NE - you also will need to RSVP.
Speakers are expected to include:
- Risa Lavizzo-Mourey, M.D., M.B.A., President and CEO, Robert Wood Johnson Foundation
- David R. Williams, M.P.H., Ph.D., Norman Professor of Public Health, Harvard School of Public Health
- Patrick Remington, M.D, M.P.H., Associate Dean for Public Health, University of Wisconsin School of Medicine and Public Health
- Andrew Webber, President and CEO, National Business Coalition on Health
- Judith A. Monroe, M.D., FAAFP, State Health Commissioner, Indiana State Department of Health
- Donald Shell, M.D., M.A., Health Officer, Prince George’s County Health Department
(Full disclosure: The Robert Wood Johnson Foundation is one of a number of organizations supporting AHCJ’s educational efforts.)
Update
See how many news organizations have covered the rankings.
Oransky: Medical study embargoes serve whom?
Embargoes, a fairly frequent topic of discussion on Covering Health, seem to bring out strong feelings in some people.
In one recent example allegedly involving embargoes, TheStreet.com’s Adam Feuerstein attempted to combat a rumor that the New England Journal of Medicine would be publishing an article about the experimental lung cancer drug seliciclib - a rumor that was seemingly driving up the stock price of Cyclacel Pharmaceuticals.
Feuerstein looked at his advance copy of NEJM and reported that no such article was coming out. He was immediately accused by a commenter on the story of having broken the NEJM’s embargo.
Photo by Billingham via Flickr
As Reuters Health Executive Editor - and AHCJ board member - Ivan Oransky asked, “Is saying what’s NOT in an embargoed journal breaking the embargo?” Oransky concluded that it was not a case of breaking an embargo. The ensuing discussion on Twitter and in the comments on Feuerstein’s article is interesting. For the record, the NEJM apparently sided with Feuerstein, according to a commenter on the story.
Today, Oransky weighs in on the wider topic of who is served by embargoes on medical studies:
Two weekends ago, at ScienceOnline2010, I heard an interesting thing about embargoes. Connie St. Louis, who directs the science journalism masters’ program at City University, London, told an audience that one of the reasons for embargoes on scientific journal studies is that with more eyeballs on the study before publication, it’s more likely researchers will catch flaws in papers, which can then be pulled.
In other words, just as the FDA requires drug companies to monitor drug side effects in large populations once a drug is approved, in what’s called post-marketing surveillance or phase 4 testing, because such side effects may not show up in relatively small trials, this is a sort of post-acceptance peer review. So if a reviewer doesn’t catch an error during the normal course of peer review, journals can use the embargo period as a backstop.
I hadn’t heard that before, and I consider myself fairly well-acquainted with the arguments for and against embargoes. But it reminded me again that for all the talk of embargoes serving the public by allowing reporters to write more-informed stories, there are serious questions about whether journals are the group that gains most from embargoes.
As it turned out, I was in the midst of another episode that reminded me of that. Last week, the Cochrane Library published their quarterly set of reviews. Among them was a review of whether opioid drugs, when used as prescribed, carried a high rate of addiction. Cochrane researchers found that they don’t. That’s not the biggest research finding ever, but when you run a health news wire service filled with dozens of stories a day, like I do, it’s something worth covering. Plus, opioid dependence remains a big issue, for celebrities and lawmakers alike.
That study was embargoed for 7 p.m. Eastern on the 19th. Around the same time I was reviewing it, the Annals of Internal Medicine press packet hit my desk. That packet — embargoed until 5 p.m. Eastern Monday the 18th — included a paper that said high doses of opioids, even if prescribed, increased the risk of overdose. Again, not the most shocking study ever, but researchers and advocates continue to debate whether these drugs, when used in prescribed doses, are dangerous.
I figured the best way to serve our readers would be a story that included both of these studies, both as context for the other. Trouble was, if I ran one based on the Annals study, I couldn’t mention the Cochrane review. And if I waited for the Cochrane review’s embargo to lift, a competitor might run the other story. (Yes, we think about these things.)
So I emailed Jennifer Beal, who runs media relations for the Cochrane Library’s publisher, Wiley, explaining the situation, and asking her if she would consider moving the embargo. She returned my message right away, saying politely that she couldn’t, and explaining why, in a message that I found thoughtful.
“We thought about it very carefully but felt that our guiding principle with embargoes is that we are giving media an opportunity to investigate a story fully without the pressure to publish immediately, so that the story is still ‘new’ on the day the research gets published, and is therefore available for public consumption,” she wrote. “If we were to agree to a moved embargo, it would mean that you…would be writing about a story where the research was not available for the public to read if they wish and make up their own minds.”
The opioid study, she noted, was “one of approximately 160 articles publishing on Wednesday; this is a big operation where the publication schedule is planned out a long way in advance, so it is not possible to move forward the publication date.” (In a long-planned move, Cochrane is now going monthly, which will distribute the reviews more evenly.)
Based on resource constraints, I decided we could only run one story on opioid addiction or overdose last week. We ended up running a story on the Annals study, which seemed a bit more newsworthy than the Cochrane review. As it turns out, a number of news organizations covered the Annals study — Seattle’s LocalHealthGuide ran an item, which the Seattle Times picked up; The Seattle Post-Intelligencer, and our competitors HealthDay and Bloomberg, among others. As far as I can tell, only a site called MedIndia ran a story by the Health Behavior News Service pegged to the Cochrane review.*
(I should say here that I’m a big fan of the Cochrane Library, access to which is an AHCJ benefit. Some have criticized them for rejecting everything other than randomized double-blinded controlled clinical trials, but I’d rather see more rigor than less, given how often hype and poor evidence tend to rule the day. So this isn’t really about a problem at Cochrane. It’s about how embargo policies, whether well-intentioned or not, often make me wonder whether they serve the public.)
So whom did this Cochrane embargo serve? I’d argue it didn’t serve the public, because we and others couldn’t include news of it in the story we did decide to run. You might even say it didn’t serve Cochrane either, since I’m guessing many of my colleagues decided not to run something on it for similar reasons. And their rationale for not moving the embargo at least had the public in mind.
The episode reminded me of the CDC-autism embargo fiasco last fall, in which the CDC and the journal Pediatrics refused to lift an embargo on autism rate data that many in the autism advocacy community had already reported on. To me, the Cochrane decision was more justifiable, even if I didn’t like it.
Many have questioned embargoes, notably Vincent Kiernan, in his 2006 book “Embargoed Science.” Kiernan makes a convincing argument that embargoes serve journals most, by giving reporters something to cover every month or every week. Nowadays, that’s even more true, I think, as I see an increase in papers embargoed just 1-2 days, rather than the typical 5-6.
Still, when you run a high-volume news service, as much as you’d like to, it’s not reasonable to reject all embargoes in favor of 100% enterprising reporting. Our clients would be very unhappy, and justifiably so. Instead, we can try to cover studies with as much skepticism and context as possible. But when I hear yet another reason why embargoes might help journals, as I did at ScienceOnline2010, I’m only encouraged more to challenge the idea that embargoes are there for the public, and at least force journals to defend how they handle them.
Ivan Oransky, M.D., is executive editor of Reuters Health and AHCJ treasurer. Follow him on Twitter, where he periodically gets into debates about embargoes, among other things.
*Update:
After this post was published, we heard from Lisa Esposito, editor of the Health Behavior News Service, who tells us that Medscape and Elsevier Global Medical News did cover the Cochrane opioid review.
Open government directive bears fruit, databases
Filed under: Government, Health data, Hospitals, Hot Health Headline, Public records, Tools
In December, 2009 Peter Orszag, director of the White House’s Office of Management and Budget, issued an Open Government Directive (original PDF here) requiring a number of agencies to “identify and publish online in an open format at least three high-value data sets” on Data.gov within 45 days. That deadline came on Jan. 22, and the resulting data sets have all been posted online. The beefiest and most immediately useful are those from the Department of Veterans Affairs, but we’ve also included other sets which could prove useful for health care journalists. Descriptions are taken directly from Data.gov.
Department of Veterans Affairs
Veterans hospital report cards and safety reports
The VA has divided report cards (11 categories) and safety reports (4 categories) into topic-specific files, from Infrastructure to Nosocomial Infections. The best way to find what you’re looking for is to visit the Open Government Directive site and then scroll down to “Department of Veterans Affairs.”
FY08 Veterans Compensation and Pension by County
The Compensation and Pension by County dataset is a count of the number of veterans receiving disability compensation or pension payments from the Department of Veterans Affairs. The data is reported at the county level, by age group and by % disability rating.
Social Security disability claims
SSA Disability Claim Data
The dataset includes fiscal year data for initial claims for SSA disability benefits that were referred to a state agency for a disability determination. Specific data elements for each year and state include receipts, determinations, eligible population, and favorable determination rates.
SSA State Agency Workload Data
The dataset includes monthly data from October 2000 onwards concerning initial claims for SSA disability benefits that were referred to a state agency for a disability determination.
USDA nutrition data
MyPyramid Food Raw Data
MyPyramid Food Data provides information on the total calories; calories from solid fats, added sugars, and alcohol (extras); MyPyramid food group and subgroup amounts; and saturated fat content of over 1,000 commonly eaten foods with corresponding commonly used portion amounts.
USDA National Nutrient Database for Standard Reference
The USDA Nutrient Database for Standard Reference, Release 22 (SR22) is the major source of food composition data in the United States and provides the foundation for most public and private sector databases. SR22 contains nutrient data for over 7,500 food items for up to 143 food components, such as vitamins, minerals, amino acids, and fatty acids.
Medicare
CY 2009 MTM Contact List
CMS approved contact list of Part D Sponsors in Medication Therapy Management Program (MTMP) which is in their plans’ benefit structure.
Office of Medicare Hearings and Appeals Claims Listed by State
Total count of Claims received by Region, State and fiscal year. Appeals can be found here.
Part B National Summary Data File
The Medicare Part B National datasets are summarized by meaningful Health Care Common Procedure Coding/Current Procedural Terminology, (HCPC/CPT), code ranges. Each dataset displays the allowed services, allowed charges and payment amounts by HCPC/CPT codes and prominent modifiers.
Other
OSHA Data Initiative - Establishment Specific Injury and Illness Rates
Each year the Occupational Safety and Health Administration (OSHA) collects work-related injury and illness data from employers within specific industry and employment size specifications. This data collection is called the OSHA Data Initiative or ODI. The data provided is used by OSHA to calculate establishment specific injury and illness incidence rates.
What else is there?
The “Tools” section of the site includes widgets and data-mining and extraction tools, applications, and other services to “provide the public with simple, application-driven access to Federal data with hyperlinks.” The “Geodata” section includes federal geospatial data with metadata and links to more detailed Federal Geographic Data Committee (FGDC) metadata information.
The site is soliciting comments about what datasets should be made available, so you can suggest more datasets here. The site also offers a tutorial.
Duo writes about how health statistics can mislead
Filed under: Europe, Health data, Health journalism, Studies, Tools
Writing in mathematics-focused Plus Magazine, Mike Pearson (bio) and David Spiegelhalter (bio|wikipedia) examine not only the variety of methods used to report health statistics, but also just how each of those methods is employed to mislead physicians, patients and journalists alike. The piece was adapted from their Understanding Uncertainty Web site. The site, which is aimed in part at helping journalists understand statistics and probability, is profiled in this story.
The duo point out and illustrate common pitfalls and summarize relevant research. Not only do they point out fundamentals such as advantages that “number needed to treat,” and to a lesser extent absolute risk (1 in 100,000), numbers have over the popular relative risk (30 percent more likely), they also go much deeper. For example:
One of the most misleading, but rather common, tricks is to use relative risks when talking about the benefits of a treatment, for example to say that “Women taking tamoxifen had about 49% fewer diagnoses of breast cancer”, while potential harms are given in absolute risks: “The annual rate of uterine cancer in the tamoxifen arm was 30 per 10,000 compared to 8 per 10,000 in the placebo arm”. This tends to exaggerate the benefits, minimise the harms, and in any case make it hard to compare them. This way of presenting risk is known as mismatched framing, and was found in a third of studies published in the British Medical Journal.
And mixing and matching numbers isn’t the only way statistics can be misleading; the writers list many. Even the humble denominator can be manipulated.
For example, people have been offered a prize for drawing a red ball from a bag, and then given the choice of two bags: one containing 1 red ball and 9 white balls, the other containing 8 red balls and 92 white balls. The majority chose the bag with 8 red balls, presumably reflecting a view that it gave more opportunities to win, even though the chance of picking a red ball was lower for this bag. Similarly, people confronted with the statement “Cancer kills 2,414 people out of 10,000,” rated cancer as more risky than those told “Cancer kills 24.14 people out of 100″. The potential influence of the size of the numerator and denominator is known as the ratio bias. Frequencies are generally used in risk communication, but it is important to keep a common denominator in all comparisons.
For a thorough primer on statistics and health, the authors highly recommend Helping Doctors and Patients Make Sense of Health Statistics (pdf), an engaging 2008 paper that makes heavy use of examples and anecdotes to illustrate key issues in the interpretation of statistics.
That paper’s authors recommend the following best practices for writing about health statistics:
We recommend using frequency statements instead of single-event probabilities, absolute risks instead of relative risks, mortality rates instead of survival rates, and natural frequencies instead of conditional probabilities.
Also of interest is this related editorial (pdf) in which media are described as “enablers” of statistical illiteracy. The author also points out that, even if journalists communicate risk in the most objective possibly fashion, folks from different cultural backgrounds will still perceive it differently. It includes an interesting side note about the far-reaching impact of how physicians are allowed to define their own legal standard of care.
Related
AHCJ tip sheets
- Statistical errors even you can find
- What you need to know about risks, rates and ratios
- Medicine 101: Words, numbers and journals
AHCJ articles
- Health Journalism 2008: Lies, damned lies and medical statistics - how to interpret the evidence
- Improving reporting on medical studies
- LINK: Finding and Using Health Statistics
MIWatch.org calls for real disclosure reform
Filed under: Conflicts of interest, Pharmaceuticals, Studies
Phyllis Vine at MIWatch.org, a site that follows news about mental illness, asks whether drug company disclosures about payments made to doctors go far enough and whether anyone actually pays attention to such disclosures.
Vine raises the question of doctors taking part in “educational settings, including grand rounds, courses at professional conferences, or continuing education programs that pharma spends billions of dollars underwriting.”
She addresses the disproportionate number of psychiatrists who represent pharmaceutical companies and dominate the upper bracket of paid speakers. Vine also notes that, while many schools have drafted or are drafting policies about faculty-industry relations, enforcement of those policies is questionable.
Read the whole post on MIWatch.org.
Report focuses on researchers’ conflicts of interest
Filed under: Conflicts of interest, Government, Studies
There are vulnerabilities in how financial conflicts of interest are handled by NIH-funded researchers, according to a report (PDF, 46 pages) released today by the Department of Health and Human Services’ Office of the Inspector General. Among the findings:
- 90 percent of the grantee institutions rely solely on the researchers’ discretion to determine which financial interests are required to be reported
- because nearly half of the grantee institutions do not require researchers to provide specific amounts of equity or compensation on their financial disclosure forms, specific financial interests of NIH-funded researchers are often unknown
- grantee institutions do not routinely verify the information submitted by researchers about their financial interests
- some grantee institutions lack documentation to support their oversight of financial conflicts of interest
- the majority of grantee institutions do not have policies and procedures that address subgrantee compliance with federal regulations regarding financial conflicts of interest
- conflicts were not reported by grantee institutions to NIH in a consistent format
- grantee institutions are not required to report to NIH any financial interests that they have with outside companies
The inspector general’s review focused on the 41 grantee institutions that submitted financial conflict-of-interest reports to NIH in fiscal year 2006.
The review found that the most common financial conflict of interest is equity ownership (including stock and stock options) in companies in which the researchers’ financial interests could significantly affect the grant research.
Other financial conflicts of interest among researchers involved inventing technology, consulting, or holding positions with outside companies. To manage financial conflicts of interest, grantee institutions often require researchers to disclose conflicts in research publications; however, grantee institutions rarely reduce or eliminate financial conflicts of interest.
The report includes a number of recommendations – that it has previously recommended – to improve the reporting of financial conflicts of interest.
-
Spend a week at the CDC studying public health, meeting sources. Apply for the AHCJ-CDC Health Journalism Fellowships today!
- Reporting on the intersection of health, environment
- Tip sheet: Covering cuts to Medicare reimbursements
- Briefing: Reporting on health reform between now and 2014
- Covering high-risk insurance pools: Tips for reporters
- Health care reform has passed: What's next?
- Health and education: Covering these intersecting beats
- Reporting on the business of health care
- Health reporting resources for journalists on state and local government beats
