I recently got a departmental e-mail soliciting feedback on an interesting issue: privatizing the peer-review system. In the April 2010 issue of the Bulletin of the Ecological Society of America, J. Fox and O.L. Petchey published a contribution entitled "Pubcreds: Fixing the Peer Review Process by “Privatizing” the Reviewer Commons".
They explain something we are all, at some level, aware of: that everyone wants to publish, but few (if any) really want to review. This leads to what they call the "tragedy of the commons," or that the people willing to review manuscripts are few and being taken advantage of because there is no incentive to review, only to publish. If this is an issue that affects you (it probably does), I strongly encourage you to give the original article a read. It's not something I think much about, but it brought up several point I had not previously considered, and brings into question how much of a "community" you really think scientists belong to.
They discuss several possible solutions to this "tragedy of the research commons" and ultimately propose to set up an online "PubCred Bank". All journals would ideally use this system, in which you earn 1 PubCred for each manuscript you review, and must pay 3 PubCreds for each manuscript you submit. The authors believe that one should review three times as many manuscripts as you submit; hence, the difference between credit and debit. Note that this is not something I am endorsing or agreeing with, but certainly find interesting... and possibly a whole lot of messy.
First of all, what defines a "submission"? Does each re-submission of the same manuscript count as a submission? Certainly one is not expected to review three papers every time you have to revise and re-submit. Or am I the only one that doesn't get accepted the first time? *shifty eyes*
What about when your manuscript is not properly formatted and you have to re-submit? (Again, not that THIS has ever happened to me. Erm.) In any case in which a manuscript is rejected without being reviewed, their suggestion is that you are refunded most of your payment, only paying 0.5 out of 3 PubCreds. As it proposes that editors are paid 0.5 PubCreds for each manuscript handled, this fee basically goes to the editor. I guess we can consider that the other 2.5 PubCreds go into the account out of which he pays his reviewers.
When there are multiple authors (when are there not?), who uses their PubCreds? The article states:
"Any author on multiauthored manuscripts should be permitted to pay part or all of the submission fee, so long as the authors collectively pay the entire fee. All that matters is that, collectively, the author(s) of each submission do enough reviewing to cover the cost that they, as a group, create in the reviewer commons."
This sounds like trouble to me. Maybe this will be the ultimate decider of all first author/last author fights between collaborating PIs... I'll give you first author if you'll pay the PubCreds!
If the field is in a slow season (I don't know if these exist, but it sounds reasonable), and you don't get many opportunities for reviews, should your own publication record be penalized for that? And if your lab is in an extremely prolific season, churning out data left and right, should you be penalized for not being able to review papers to keep up with your burst of publication needs?
Will we run into issues of editor favoritism, where only his or her BFFs get reviews, so only those people get to publish science? The article addresses a similar issue, that of reviewers that have been historically poor and have been "blacklisted" by editors. The suggested solution is to choose co-authors whose PubCreds you can use. That. Does Not. I don't think- What? This seems like 7 kinds of bad idea. (Technically, it says "...blacklisted individuals would have to rely on PubCreds earned by co-authors." Potato Potahto.)
Will this lead to harried, crappy reviews that people pump out just so they can get their own publications out? There are already plenty of bad reviewers out there... I don't think the system needs any more incentive to decrease quality, even if it is in favor of quantity. Unfortunately, a devil of the system is that if an incompetent reviewer is also a slow reviewer, the editor feels obligated to get a response back to the authors quickly (instead of possibly soliciting a better reviewer), which I think leads to good papers being rejected for want of better reviews. The article addresses this by saying:
Reviewers providing late, superficial, sloppy, or inappropriate reviews should receive no PubCreds for doing so. The handling editor would decide whether a review was too late, superficial, or sloppy to be useful, and therefore to earn a PubCred. We believe that most handling editors are sufficiently frustrated by the frequent provision of extremely brief, cursory reviews that they would have no hesitation in refusing credit for such reviews.
That is, if they make it to the review stage in the first place, which it seems many good papers do not; editor overload is a main issue the PubCred system seeks to address. But again, the power of the editor in the previous excerpt makes me leery.
They also discuss the proposition of re-using reviews when you've been rejected from a journal, and are submitting to a second journal. This would keep "costs" down, as you would not have to "pay" to get the same manuscript reviewed a second time. How practical is it for journals to share reviews? I see two potential problems with this: 1) Do you want journals knowing that your article was already rejected by another journal? Do you want them to know which journal it was, especially if it was a lower-tier journal? Or that their journal was not your first pick? How is this going to influence editors when making final decisions? and 2) Some reviews are written in light of the journal's specific audience, or the type of papers it has historically accepted. I'm not sure how helpful a journal-specific review would be to a different journal, possibly with a very different audience and expectations. In that same vein, who submits exactly the same manuscript to two different journals without at least trying to give it an angle more palatable to the second journal?
Is it practical to expect all journals, from Nature to Copeia, to subscribe to the same system?
As was made painfully clear to me by my commenters on The Scoop and the re-posting at DrugMonkey, science is competitive. It seems like the drive to conceive science, do science, write science, and publish science should feel more like one long push for each project than four little ones. Is it unfair to put a roadblock in between the writing and publishing stages? How does it affect our use of publication record as a metric of academic success if reviewing other people's work (three times as much) is a prerequisite? Because reviewing is a vital contribution to the field, should this even be a problem?
Most importantly, do PubCreds also work at PUBS? Oh I hope so.
The authors have openly solicited feedback, so feel free to contact them if you are so inclined. If you are interested, the authors have also set up an online petition. I find both the problems and solutions worthy of our time, and I'm sure there is no perfect solution. But is there a better one than this?
Updated 07/26/2010 to include a couple of links I forgot the first time.
Subscribe to:
Post Comments (Atom)
Thanks for posting. I can clarify a couple of minor points and comment on a couple of others (can't take the time to comment on all of them right now...):
ReplyDelete-A "submission" is whenever you submit a new ms; an invited revision doesn't count. But if you're rejected by one journal and you revise and resubmit to a second journal, that revision is indeed a new submission. After all, the second journal is going to treat it as such, so it should be paid for as such.
-I agree that there are concerns if editors keep asking the same small pool of "usual suspects" for lots of reviews. But as an editor myself I can reassure you that while this does happen, it's not because editors ask for reviews from their BFF's. Editors tend to ask for reviews from (i) people who they know will say yes and do a good job (which often includes their close personal friends, but also includes lots of other people), (ii) people who've recently published on the same topic (especially in the journal the editor edits for), (iii) people who are famous for studying the same topic, and possibly (iv) senior students and postdocs in the labs of people in category (iii). I agree that it would be a good thing if editors were to cast a wider net when seeking reviewers, and the PubCred system could facilitate that.
-The idea of sharing reviews among journals isn't part of the PubCred system, but is rather a possibly-complementary idea that was suggested by Hochberg et al. 2009. It's an idea that needs a lot more thought and discussion. One possibility is to make it an optional choice on the part of the author, which under the PubCred system could allow submission at a discount if the editor deems that the previous reviews reduce or eliminate the need for further reviews.
-The issue of which and how many journals need to participate for the system to work is a tricky one. Ideally, I think you want most or all of the journals that comprise a "field", so that most people are either happy to confine most or all their reviewing and submitting to journals in that field, or happy to confine most or all their reviewing and submitting to other journals. Owen and I hope that "ecology", or perhaps "ecology and evolution" is sufficiently close to being a "field" for PubCreds to work. After all, if you have to get every journal in biology, or in all of science, to participate, you'll never get the system set up in the first place! We're going to work with Stefano Allesina, a theoretical ecologist, to try to address this and other issues with simulations. Stefano has an NSF grant to model alternative peer review systems. I don't claim that models will provide all the answers, but they're better than nothing.
-The hope is that allowing overdrafts on PubCred accounts will allow people to get through periods when they're writing a lot of papers in a short period of time.
-It's the author's fault if an ms is rejected for improper formatting! Yes, this is a trivial reason for rejecting a paper--but it's also trivially easy to avoid (if you think it's not, I think you're in *way* too much of a hurry to submit your work...). There's a broader issue here, though, which is that journals increasingly look for any excuse to reject a paper. If the PubCred system makes willing and conscientious reviewers easy to find, journals will have much less reason to reject papers for trivial reasons.
-Yes, editors have a lot of power in the PubCred system. But they have a lot of power in the current system. The question is how that power is used. An "editor's code of conduct" might be of some value here.
-I completely share the sentiment of your final two sentences. Well said!
You're the best
ReplyDelete