Scrutiny of Science

GOING ASTRAY? In their enthusiasm for perfect results, some researchers resort to methods that are not quite acceptable. Benedict Carey finds out what the consequences are.

The crimes and misdemeanors of science used to be handled mostly in-house, with a private word at the faculty club, barbed questions at a conference, maybe a quiet dismissal. On the rare occasion when a journal publicly retracted a study, it typically did so in a cryptic footnote. Few were the wiser; many retracted studies have been cited as legitimate evidence by others years after the fact.

But that gentlemen’s world has all but evaporated, as a remarkable series of events last month demonstrated. In mid-May, after two graduate students raised questions about a widely reported study of the effect of political canvassing on opinions of same-sex marriage, editors at the journal Science, where the study was published, began to investigate.

What followed was a frenzy of second-guessing, accusations and commentary from all corners of the Internet: “Retraction” as serial drama, rather than footnote. Science officially pulled the paper, by Michael LaCour of the University of California, Los Angeles, and Donald Green of Columbia, on May 28, because of concerns about Michael’s data.

“Until recently it was unusual for us to report on studies that were not yet retracted,” said Dr Ivan Oransky, an editor of the blog Retraction Watch, the first news media outlet to report that the study had been challenged. But new technology and a push for transparency from younger scientists have changed that, he said. “We have more tips than we can handle.”

The case has played out against an increase in retractions that has alarmed many journal editors and authors. Scientists in fields as diverse as neurobiology, anaesthesia and economics are debating how to reduce misconduct, without creating a police-state mentality that undermines creativity and collaboration. “It’s an extraordinary time,” said Brian Nosek, a professor of psychology at the University of Virginia, and a founder of the Center for Open Science, which provides a free service through which labs can share data and protocols. “We are now seeing a number of efforts to push for data repositories to facilitate direct replications of findings.”

But that push is not universally welcomed. Some senior scientists have argued that replication often wastes resources. “Isn’t reproducibility the bedrock of science? Yes, up to a point,” the cancer biologist Mina Bissell wrote in a widely circulated blog post. “But it is sometimes much easier not to replicate than to replicate studies,” especially when the group trying to replicate does not have the specialised knowledge or skill to do so.

Digging for dope
The experience of Retraction Watch provides a rough guide to where this debate is going and why. Dr Ivan, who has a medical degree from New York University, and Adam Marcus, both science journalists, discovered a mutual interest in retractions about five years ago and founded the blog as a side project. They had, and still have, day jobs: Marcus, 46, is the managing editor of Gastroenterology & Endoscopy News, and Oransky, 42, is the editorial director of MedPage Today (he will take a position as distinguished writer in residence at NYU later this year).

In its first year, the blog broke a couple of retraction stories that hit the mainstream news media - including a case involving data faked by an anaesthesiologist who later served time for health care fraud. The site now has about 150,000 unique visitors a month, about half from outside the United States.

Dr Ivan and Adam are partisans who editorialise sharply against poor oversight and vague retraction notices. But their focus on evidence over accusations distinguishes them from watchdog forerunners who sometimes came off as ad-hominem cranks. Last year, their site won a $400,000 grant from the John D and Catherine T MacArthur Foundation, to build out their database, and they plan to work with Brian to manage the data side.

Their data already tell a story. The blog has charted a 20 to 25 per cent increase in retractions across some 10,000 medical and science journals in the past five years: 500 to 600 a year today from 400 in 2010. (The number in 2001 was 40, according to previous research.) The primary causes of this surge are far from clear. The number of papers published is higher than ever, and journals have proliferated, Ivan and other experts said.

New tools for detecting misconduct, like plagiarism-sifting software, are widely available, so there’s reason to suspect that the surge is a simple product of better detection and larger volume.

Still, the pressure to publish attention-grabbing findings is stronger than ever, these experts said - and so is the ability to “borrow” and digitally massage data. Retraction Watch’s records suggest that about a third of retractions are because of errors, like tainted samples or mistakes in statistics, and about two-thirds are because of misconduct or suspicions of misconduct.

The reason being...
The most common reason for retraction because of misconduct is image manipulation, usually of figures or diagrams, a form of deliberate data massaging or, in some cases, straight plagiarism. In their dissection of the LaCour-Green paper, the two graduate students - David Broockman, now an assistant professor at Stanford, and Joshua Kalla, at California-Berkeley - found that a central figure in Michael’s analysis looked nearly identical to one from another study. This and other concerns led Donald, who had not seen any original data, to request a retraction. (Michael has denied borrowing anything.)

Data massaging can take many forms. It can mean simply excluding “outliers” - unusually high or low data points - from an analysis to generate findings that more strongly support the hypothesis. It also includes moving the goal posts: that is, mining the data for results first, and then writing the paper as if the experiment had been an attempt to find just those effects. “You have exploratory findings, and you’re pitching them as ‘I knew this all along,’ as confirmatory,” Brian said.

The second leading cause is plagiarising text, followed by republishing - presenting the same results in two or more journals. The fourth category is faked data. No one knows the rate of fraud with any certainty. In a 2011 survey of more than 2,000 psychologists, about one percent admitted to falsifying data. Other studies have estimated a rate of about two percent.

Yet one offender can do a lot of damage. The Dutch social psychologist Diederik Stapel published dozens of studies in major journals for nearly a decade based on faked data, investigators at the universities where he had worked concluded in 2011. Suspicions were first raised by two of his graduate students.

“If I’m a scientist and I fabricate data and put that online, others are going to assume this is accurate data,” said John Budd, a professor at the University of Missouri and an author of one of the first exhaustive analyses of retractions, in 1999. “There’s no way to know” without inside information. “At this point, we see ourselves as part of an ecosystem that is advocating for increased transparency,” Dr Ivan said. “And that ecosystem is growing.”