Refereeing in Philosophy Pt. 4

Welcome to the fourth and final part of my series on the referee crisis in philosophy. If you want background on refereeing in general, read part 1. If you want background on the crisis, read part 2. If you want to see what journals have actually done to combat the issues, read part 3. This (final) installment is about solutions that have not been implemented (as far as I know), including some of my own.

In a nutshell, the core of the issue is that the need for refereeing, which is a vital part of research/publication, exceeds the system’s ability to use the available supply. There are two important reasons for this. Number one: freeriders, who generate more work in submissions than they contribute in reports. Number two: inefficiencies in how journals find their referees. There’s no good way for editors to know who is available when. As a result, some people get asked more than they can reasonably do, and others who might be able to do more simply don’t get asked enough.

We’ll begin with some proposals out there that have been floating around for a while. The first one is very similar to what has already been done: it uses submission limits to cut down the number of papers journals have incoming. David Vellemen gives an example of it here. The gist of this idea is to limit submissions by career stage. Vellemen’s version of the proposal is the best known, where he proposes journals not accept manuscripts from grad students. In the intervening years, we have seen some undergrads publishing major journals, and correspondingly some discussion of ways to discourage undergrad publishing.

The main argument for these anti-student-publishing proposals is two-fold. First: competition for grad school places or a first job is fierce enough without normalizing pre-doctoral publication. Removing the burden of publishing for students lets them focus on becoming better thinkers/researchers, and allows them to enjoy their precious grad school years developing their networks and interests. Second: the average quality of student submissions is lower than that of those at more advanced career stages, so eliminating them both cuts down the submission pool and cuts down on manuscripts particularly at risk for repeat submission in search of the right referee luck.

I am not a fan of these proposals, to a point. As much as publication plays a role in career advancement, that is not its primary function. Its primary function is to get the best research papers published. The identity of the author shouldn’t be very important in that. There is some loss in slower-developing thinkers being filtered out at early stages such as the Ph.D admission or first job stage, but whether it outweighs the loss of slowing down other thinkers who have publication-ready work is unclear. As we will see, a proposal I do endorse would have a side-effect of cutting down on student submissions. But it wouldn’t be as crude as an outright ban.

The next proposal is to have “enduring” referee reports. These would follow papers around, so that each time it is submitted to a new journal some or all of the old reports go with it. This kind of happens in ‘transfer’ ecosystems, where a publisher gives editors the opportunity to transfer a rejected paper to a different journal that publisher publishes. In these cases, it is up to a referee whether their report should accompany the paper if transferred. This would reduce the demand for referee reports, but at the price of vastly inflating the power of a paper’s initial referees. Given the wide variance in referee quality and judgment, I think this is another bad idea. New submissions deserve new eyes. In fact, I usually will not referee the same paper for different journals – it’s one of the main reasons I will turn an assignment down.

A final proposal that tries to raise the supply of referee reports advocates for compensating referees. This is already standard practice for referee reports for book proposals. Publishers usually over a few hundred dollars in credits for books in their catalog, or a discount on books from their catalog. But it is basically unheard of for journal referees. There’s a good reason for this; at rates that would actually provide an incentive, the cost would be prohibitive. But there have been other proposals to incentivize refereeing, such as this one for a database where referees can receive public credit. I like the idea of attaching more rewards to referee work, although my preferred version is different from monetary or public credit.

There are other ideas out there; this Philosophers’ Cocoon post has a good quick summary with links. I’m going to focus the rest of this essay on some solutions that so far as I know are novel.

One observation about the solutions proposed and implemented so far: they largely focus on narrowing the gap between supply and demand of referee reports by reducing the volume of submissions. There have been a few proposals to increase the supply of referee reports, but I don’t think any of them would fully bridge the gap.

An underlooked gap in the system, I think, is the inefficiency in editors seeking referees. Most editors have a process like the following: after they read a paper, they ask people in their network who they think would be good. If no one bites, and they run out of suggested referees from their declines, they go to databases like philpapers (as suggested here) or look at authors in the paper’s footnotes to find more people. This makes it easy for people to fall through the cracks, or for editors to ask people at the wrong time. There has been a recent attempt at further improvement, in the form of a public spreadsheet where people can indicate their availability. But a google sheet is a clunky interface, and it relies on people knowing about it and regularly updating their status.

It would be better if available philosophers could volunteer to referee papers. It’s much easier for a journal (or journals) to maintain a viewable database of submissions than for the profession as a whole to maintain a database of available referees. And willing referees would not have to wait to be asked in order to contribute; they could simply find a paper that looked like a good fit and volunteer. This may not eliminate all of the inefficiencies in the referee search process, but it would take a big bite out of them.

Of course, the details matter. Tentatively, I suggest the following: once a paper has passed desk review at a journal, it is added to the database with a number, subdisciplines, and keyword tags from a standard menu such as the one philpapers offers. This will preserve anonymity and prevent any gaming of the system by friends/associates. Registered users (with some sort of screening criteria, as a first pass a Ph.D from a recognized institution) can then search by topic for papers within their specializations. They can then volunteer to referee a paper via a small form indicating their expected completion time-frame. This sends a message to the editor who posted it, who can then see if the volunteer fits their criteria for a referee for the paper. If the stars align, the referee receives the paper. If not, it stays up. Once the journal has enough referees lined up, the paper goes off the database.

This would make the search for referees much easier. But why would referees volunteer? Some may because of intrinsic features of the work, or in order to influence the development of the literature. But an additional incentive would be good. My suggestion: refereeing more papers should earn people more journal submissions. Perhaps by default, journals allow one submission per year. This will make room for people who cannot referee enough for legitimate reasons (e.g. they are too junior, doing other service work, or have various life circumstances such as illness or hardship). But in order to submit to the journal more than once per year, someone must generate enough referee reports for them to offset the ones they generate (plus some, to make up for the free submission everyone gets). A natural number here is three. It would probably also be good for journals to allow credit sharing, so that refereeing for Journal A can count toward an extra submission at Journal B, but this is the kind of agreement that journals could negotiate among themselves voluntarily.

I’m still somewhat tentative on these proposals, although I’m more confident that the volunteer system is a good idea than I am that linking referee work to ability to generate extra submissions is. Rather than try to run through a bunch of objections, I will it to the commenters to object and address them as they arise.