Scientific Publishing Is a Joke

A real scientific advance, like a successful date, needs both preparation and serendipity. As a tired, single medical student, I used to feel lucky when I managed two good dates in a row. But career scientists must continually create this kind of magic. Universities judge their research faculty not so much by the quality of their discoveries as by the number of papers they’ve placed in scholarly journals, and how prestigious those journals happen to be. Scientists joke (and complain) that this relentless pressure to pad their résumés often leads to flawed or unoriginal publications. So when Randall Munroe, the creator of the long-running webcomic XKCD, laid out this problem in a perfect cartoon last week, it captured the attention of scientists—and inspired many to create versions specific to their own disciplines. Together, these became a global, interdisciplinary conversation about the nature of modern research practices.

The cartoon is, like most XKCD comics, a simple back-and-white line drawing with a nerdy punch line. It depicts a taxonomy of the 12 “Types of Scientific Paper,” presented in a grid. “The immune system is at it again,” one paper’s title reads. “My colleague is wrong and I can finally prove it,” declares another. The gag reveals how research literature, when stripped of its jargon, is just as susceptible to repetition, triviality, pandering, and pettiness as other forms of communication. The cartoon’s childlike simplicity, though, seemed to offer cover for scientists to critique and celebrate their work at the same time.

The concept was intuitive—and infinitely remixable. Within a couple of days, the sociologist Kieran Healy had created a version of the grid for his field; its entries included “This seems very weird and bad but it’s perfectly rational when you’re poor,” and “I take a SOCIOLOGICAL approach, unlike SOME people.” Epidemiologists got on board too—“We don’t really have a clue what we’re doing: but here are some models!” Statisticians, perhaps unsurprisingly, also geeked out: “A new robust variance estimator that nobody needs.” (I don’t get it either.) You couldn’t keep the biologists away from the fun (“New microscope!! Yours is now obsolete”), and—in their usual fashion—the science journalists soon followed (“Readers love animals”). A doctoral student cobbled together a website to help users generate their own versions. We reached Peak Meme with the creation of a meta-meme outlining a taxonomy of academic-paper memes. At that point, the writer and internet activist Cory Doctorow lauded the collective project of producing these jokes as “an act of wry, insightful auto-ethnography—self-criticism wrapped in humor that tells a story.”

Put another way: The joke was on target. “The meme hits the right nerve,” says Vinay Prasad, an associate epidemiology professor and a prominent critic of medical research. “Many papers serve no purpose, advance no agenda, may not be correct, make no sense, and are poorly read. But they are required for promotion.” The scholarly literature in many fields is riddled with extraneous work; indeed, I’ve always been intrigued by the idea that this sorry outcome was more or less inevitable, given the incentives at play. Take a bunch of clever, ambitious people and tell them to get as many papers published as possible while still technically passing muster through peer review … and what do you think is going to happen? Of course the system gets gamed: The results from one experiment get sliced up into a dozen papers, statistics are massaged to produce more interesting results, and conclusions become exaggerated. The most prolific authors have found a way to publish more than one scientific paper a week. Those who can’t keep up might hire a paper mill to do (or fake) the work on their behalf.

In medicine, at least, the urgency of COVID-19 only made it easier to publish a lot of articles very quickly. The most prestigious journals—The New England Journal of Medicine, the Journal of the American Medical Association, and The Lancet—have traditionally reserved their limited space for large, expensive clinical trials. During the pandemic, though, they started rapidly accepting reports that described just a handful of patients. More than a few CVs were beefed up along the way. Scientists desperate to stay relevant began to shoehorn COVID-19 into otherwise unrelated research, says Saurabh Jha, an associate radiology professor and a deputy editor of the journal Academic Radiology.

A staggering 200,000 COVID-19 papers have already been published, of which just a tiny proportion will ever be read or put into practice. To be fair, it’s hard to know in advance which data will prove most useful during an unprecedented health crisis. But pandemic publishing has only served to exacerbate some well-established bad habits, Michael Johansen, a family-medicine physician and researcher who has criticized many studies as being of minimal value, told me. “COVID publications appear to be representative of the literature at large: a few really important papers and a whole bunch of stuff that isn’t or shouldn’t be read,” he said. Peer-reviewed results confirming that our vaccines really work, for example, could lead to millions of lives being saved. Data coming out of the United Kingdom’s nationwide RECOVERY trial have provided strong evidence for now-standard treatments such as dexamethasone. But that weird case report? Another modeling study trying to predict the unpredictable? They’re good for a news cycle, maybe, but not for real medical care. And some lousy studies have even undermined the treatment of COVID-19 patients (hydroxychloroquine has entered the chat).

I should pause here to acknowledge that I’m a hypocrite. “Some thoughts on how everyone else is bad at research” is listed as one of the facetious article types in the original XKCD comic, yet here I am rehashing the same idea, with an internet-culture angle. Unfortunately, because The Atlantic isn’t included in scientific databases, publishing this piece will do nothing to advance my academic career. “Everyone recognizes it’s a hamster-in-a-wheel situation, and we are all hamsters,” says Anirban Maitra, a physician and scientific director at MD Anderson Cancer Center. (He created a version of the “12 Types” meme for my own beloved field: “A random pathology paper with the phrase ‘artificial intelligence’ in the title.”) Maitra has built a successful career by running in the publication wheel—his own bibliography now includes more than 300 publications—but he says he has no idea how to fix the system’s flaws. In fact, none of the scientists I talked with could think of a realistic solution. If science has become a punch line, then we haven’t yet figured out how to get rid of the setup.

While the XKCD comic can be read as critical of the scientific enterprise, part of its viral appeal is that it also conveys the joy that scientists feel in nerding out about their favorite topics. (“Hey, I found a trove of old records! They don’t turn out to be particularly useful, but still, cool!”) Publication metrics have become a sad stand-in for quality in academia, but maybe there’s a lesson in the fact that even a webcomic can arouse so much passion and collaboration across the scientific community. Surely there’s a better way to cultivate knowledge than today’s endless grid of black-and-white papers.