Disincentivizing Academic Fraud

A careful look at the systemic incentives that lead to academic fraud from the perspective of the various stakeholders in the process

You may have seen the news that Marc Tessier-Lavigne, the President of Stanford University, resigned after many of his papers were found to contain manipulated data. Following an award-winning piece of investigative journalism that called out Tessier-Lavigne for manipulation of data in high-profile publications, a Stanford review board concluded that while the papers were problematic, Tessier-Lavigne himself was either unaware of or not the source of the issues. Nevertheless, he has resigned, and will be retracting or correcting at least some of these papers.

I am not going to comment on the findings of the review board (though I will comment on this sort of review board generally). They have access to more information than I do, and so I will take their conclusion at face value as the premise for the rest of this commentary.

Tessier Lavigne is a highly decorated neuroscientist whose name is on some of the most influential papers in neurodegenerative disease the past few decades. He has been president of two universities (Rockefeller and Stanford), sits on multiple boards, and is himself a billionaire through his ownership of stock in Regeneron Pharmaceuticals. In other words, this is a man who understands how to navigate the system in which he operates, and has benefited tremendously from that system.

The conclusion of the review board was that the integrity issues stemmed from others, but when you are in a leadership position and have final say in what gets published, you are ultimately responsible for the output of your team. This only becomes more true as your profile rises, and Tessier-Lavigne had about the highest profile that an academic researcher can have. Combined with the fact that this appears to have been an issue across more than one research group he led over a span of decades, it is clear that he is the common denominator.

If it is not clear from the fact that “incentives” is probably the most-used word on this blog, I am a big believer in the old economic maxim “people respond to incentives”. In academia, there are many intersecting problematic incentives that together create conditions where this sort of outcome is essentially inevitable, and highly unlikely to be an isolated case.

Academic Publication and Peer Review

Most fields of science are discovering that they have a reproducibility crisis, wherein large fractions of previously published and peer-reviewed papers are at best wrong, and at worse, made up. The source of the problem boils down to “publish or perish” culture. If you combine endless pressure to publish as many papers as possible combined with a need to do so in high-impact journals that require novelty (no replication studies or publication of negative results) to even get past the editor, it should be no surprise that the end results is low-impact papers, irreproducible results, and outright fraud.

One might think that peer review would serve to filter out bad science, but once again, the incentives are misaligned so as to make this ineffective. There are disincentives in place that prevent this from coming from the peers who, according to the peer-review system, are the ones who should be raising criticism. Because the pace of publication of papers far exceeds human capacity to keep up, citation networks shrink to the point that there is no such thing as truly anonymous review. Calling out the work of someone else in your network (especially someone as powerful as MTL) has no upside and is a massive career risk: they review your papers, your grant applications, and chair the hiring committees with whom you or your students will interact. It comes as no surprise to me to learn that researchers in MTL’s field were well aware of potential issues and said nothing. Pubpeer has a few comments on MTL’s papers, but you will note that all those that predate his resignation are made anonymously. It is telling that the reason MTL was caught was because someone outside his citation network investigated.

Tsuyoshi Miyakawa writes about his challenges getting authors to provide raw data as a journal editor, suggesting that a raw data requirement is a very effective filter for bad science. Metrics relating to quantity of papers need to give way to those relating to quality of data, and blind, private peer review of published interpretations of data needs to give way to public and open peer verification of those interpretations starting from raw data and original analysis code, with reviewers rewarded for finding and fixing errors in published work. Moving toward open datasets and reviewed and tested analysis would go a long way toward incentivizing the right behaviors.

Journals

With researchers incentivized to chase metrics that put quantity before quality, journals are happy to charge increasing sums of money for authors to publish and charge for access, all based on the free, metric-incentivized work of authors, editors, and peer reviewers.

For journals, a retraction is an embarrassment, an acknowledgement that the review process failed, and can take literally decades (see the link in the comments). The Retraction Watch article shows that while retractions are on the rise, they are still likely only a tiny fraction of papers, and are lip service to the scope of the problem that is a fueling reproducibility crisis in all fields of science. There is really only one incentive for a journal to go through with a retraction, and it boils down to reputation management: Scientific America summarizes it nicely:

“A more cynical view—one we espouse—is that publishers are making a big deal out of such episodes only because they can paint themselves as victims of sophisticated wrongdoers. That narrative of course omits the fact that publishers poured as much gasoline as they could find on the publish-or-perish fire, and it threatens to distract us from what might be more consequential fraud.”

There have been a few attempts at addressing the myriad issues around journals over the years. Many are moving toward open access models, but open access costs extra. Some journals have data availability requirements - but if you have ever tried to get the data you will find that if it is truly available at all, it is rarely in useful form, nor is it accompanied by the code or protocol by which the authors processed it. Tools like The PubPeer Foundation allow unsolicited flagging of issues in published work. All of these are positive initiatives, but incremental fixes. No systemic problem can be fixed until the incentives that caused it are addressed, and the required change won’t come from incumbent publishers.

Academic Institutions

Most of the public-facing decisions of universities are driven by ranking systems, which are in turn based in large part on research- and impact-related metrics. These are self-fulfilling prophecies: top-ranked universities get the most funding and are able to attract the most talented researchers, which in turn results in higher rankings and concentrations of funding a talent in a few top universities. There is no ranking system of which I am aware that incentivizes prosecution of academic fraud by faculty members. For university administrators, conducting investigations into the potential misconduct of their researchers is an embarrassment.

If you look at institutional responses to high-profile cases of academic fraud, you will find that most of them have in common that public revelation from outside of the citation network of the perpetrator was a precursor to action, with institutions often preferring to actively suppress the the story where possible. As with journals issuing a retraction, academic institutions generally only convene investigative committees as a form of reputation management when issues become sufficiently public. For every case we see, there are likely many that are ignored. In the case of MTL, action comes far too late, as he has had decades to influence his field, and to export his approach to academic integrity and research oversight to his students.

Just as researchers chasing the h-index results in over-publishing, university rankings that prioritize research output and reputation incentivize universities to turn a blind eye to integrity issues. With MTL, an entire generation of experts in the field were trained by him, built on his results, and published papers that cite him. Without effective and positively incentivized institutional safeguards in place to ensure academic integrity without public revelation, integrity issues can spread, as years of work come into question and a new generation of scientists learn from an outwardly shining example that academic integrity and robust results are secondary to reputation and perceived impact - and that institutions will protect and even continue to employ you if you get caught, as long as you are a big enough name.

I don’t see a clean way to align incentives at the level of institutions. The only viable solution that I can see is to shift the focus on vetting research output toward open data, crowd-sourcing review of source data and short-circuiting institutional oversight.

What can be done?

Incentives in the current academic publication system run counter to the basic self-correcting nature of the scientific method. If the incentive is to produce a large quantity of publication over a few quality ones, to chase novelty over replication, to allow problematic results past peer-review for the sake of not angering people like MTL, or to turn a blind eye to or even suppress evidence of academic fraud for the sake of avoiding the embarrassment, it should not be a surprise that we are seeing a widespread reproducibility crisis, fragmented patchworks of citation networks, and more than one case of high-profile fraud.

You can't fix a system with broken incentives, and systemic change rarely comes from the inside. A fundamental change to the way we approach dissemination of science is needed.

The LK-99 hype provides an interesting contrast: a study reported groundbreaking results, which quickly went viral and inspired a host of reproduction attempts that eventually debunked the original claim (though I should be crystal clear in this case that there is no evidence of which I am aware of any fraud in the original study - just a mistake).

Hype cannot be the basis for peer review, but the LK-99 replication attempts by teams all over the world is how science is supposed to work, and provides an excellent proof of concept for the crowd sourced model of peer review.

There’s a policy incentive that could be used to push science in this direction. Publicly funded research often comes with a data sharing requirement, but this is almost never enforced, and data rarely made available in useful form. By tying the money to useful data sharing, we can incentivize the right behaviors.

Version-controlled raw data and analysis code provides a path for error correction and makes it more likely that fraudulent data will be caught. It reduces the pace of publication, since citation credit can go to the data itself. It allows new science to build on old data as analytics methods improve. And it addresses the mass of academic dark data that is currently being wasted.