Showing your working (papers)

While eating my breakfast this morning I came across a tweet about working papers, arguing that, effectively, we shouldn’t talk about them. 

Working papers, also known as pre-prints, are versions of an academic paper that are released, usually online, and usually for free, by researchers ahead of their publication through peer review. They allow scholars to get feedback on their research; to stake a claim on a particular finding or topic; and contribute to the discussion in the field. The tweet argues that researchers shouldn’t publicise their working papers, or talk to the media about them. 

Now, the tweet didn’t rule out speaking to policy makers, but it seemed implicit – if you can’t talk about the findings of your new dynamic stochastic general equilibrium model with a journalist (who almost certainly wouldn’t be interested), then you can’t talk to an economist at the Bank of England, who might be. After all, the embargo on publicising your research seems to be that you could have an impact that you didn’t deserve, not anything about journalists in particular.

As both an academic and former civil servant, I have to disagree. Others will argue that peer review isn’t exactly flawless in itself (see, for example, the much publicised replication crisis in social psychology), and that many pieces of rubbish research can get over that hurdle but that isn’t my point here. Instead, I’m arguing that many pieces of really good research don’t get over the threshold of peer review. 

Why does this happen? First, lots of good research is done by graduate students – PhD students and post docs. Their research might be extremely relevant for policy – but they may decide that the rat race of academia isn’t for them, and find themselves moving into consulting, government, or some other role that doesn’t afford them the (considerable) time it takes to get things published. Should the fruits of their research wither on the vine, never to be used? 

Second, the correlation between research that makes for a good academic paper, and that which can and should influence policy, is not strong. The most impactful piece of research I’ve ever conducted was a stepped wedge trial in job centres in Essex, when I worked at the Behavioural Insights Team. Our interventions were light, and seemed to have an effect; resulting in real policy change. Five years later, the paper came out in the journal of public policy after several rounds of heavy revisions. The paper had previously been rejected because we were testing a bundle of interventions, rather than one tightly defined theory. Important in an academic context, but irrelevant for the policy makers and job seekers that our research affected. 

The other factor is publication bias. While academics and other practitioners of ‘evidence based policy’ will say that null results are just as important as positive findings, in reality it’s still much harder to get them published. Finding that something doesn’t work isn’t as ‘cool’, or ‘novel’, and editors and reviewers alike, especially at top journals, still reject papers. Yet in government, the fact that an intervention doesn’t work is just as important, as it tells us what we should stop doing. 

Individual academics have their careers to look out for, and work in a system that is skewed towards rewarding publications in top journals. As such, if they fail to publish in those outlets, or suspect that they will, it is individually sensible of them to shelve that paper. I can’t blame them for doing so. But publishing working papers, subjecting them to scrutiny, and talking about them, lets the rest of us benefit from the enormous amount of work it took to get the work to that stage.

Dr Michael Sanders is Chief Executive of What Works for Children’s Social Care, and a Reader in Public Policy at the Policy Institute, King’s College London.