Should PLoS ONE count as peer-reviewed?

August 23rd, 2007 Administration 37 Comments
By Dave Munger

Bora Zivkovic’s job is to try to get people to comment on articles in PLoS ONE, the new online journal designed to get articles in front of the public quickly.

According to their journal information page, an important part of their peer review process is community review. Indeed, the journal only requires review by a single editor before publication. One commenter on Zivkovic’s blog post about the process suggests that this is an inadequate level of peer review:

My current view is that with PLoS ONE, if you have $1250, you have a published paper.

Zivkovic’s response is that all articles are fully reviewed, that reviewers don’t know whether the publication fee has been waived, and that half of all submissions are rejected, while many are revised several times before publication.

While that may be true, PLoS ONE would not qualify as peer-reviewed under the standards I’ve proposed for BPR3 (a minimum of two reviewers in addition to the journal editor prior to publication). Yet one commenter suggests that even those standards “widen” the concept of peer review.

As more journals try out the concept of community review of publications, will the concept of “peer review” become even more difficult to define? How will BPR3 fit in in an increasingly ambiguous academic world? Share your thoughts with us in the comments section.

37 Responses to “Should PLoS ONE count as peer-reviewed?”

  1. Jeremy Clark Says:
    August 23rd, 2007 at 9:18 am

    “a minimum of two reviewers in addition to the journal editor”

    Are conference submissions thus ruled out? In my discipline (electrical engineering/computer science), the most interesting work is done at peer-reviewed workshops and conferences. It may eventually find itself in a journal, but usually you want to blog about it when its fresh.

    Now that I read the other thread, I suppose this could be caught under the “other form as appropriate for the field of study.”

  2. Bora Zivkovic Says:
    August 23rd, 2007 at 9:21 am

    Over the past few weeks I had to look at every manuscript in the pipeline at PLoS ONE and each of referees’ comments. Each paper was reviewed by 2 or 3 people and their comments were as thorough, detailed and nit-picky as anything I’ve seen elsewhere. A paper in my field which, if thoroughly rewritten, I’d let through, was rehected after three rounds of reviews, each time by 2 different reviewers. Just sayin’….

  3. Dave Munger Says:
    August 23rd, 2007 at 9:27 am

    Jeremy –

    In psychology (the field I’m familiar with) peer review for conference presentations isn’t very rigorous — usually just the abstract is reviewed, and no one checks to make sure the actual presentation corresponds to the abstract. I can’t speak for engineering/comp sci, but to my mind this doesn’t cut it.

    Bora –

    It sounds like a pretty thorough job is done in the case of PLoS ONE, but this doesn’t match up with the stated policy on the site. Maybe you ought to consider revising the stated policy, if in fact your actual review process is so rigorous.

  4. Bora Zivkovic Says:
    August 23rd, 2007 at 9:30 am

    I will be talking with the Managing Editor about this very thing later today as well as at the weekly meeting of the PLoS ONE production group next week.

  5. Dr Shock Says:
    August 23rd, 2007 at 10:13 am

    Peer reviewed means at least two reviewers and the editor to my opinion with re-evaluation if the reviewers come to a draw.
    When money is involved other arguments come into play.

  6. Chris Surridge, Managing Editor PLoS ONE Says:
    August 23rd, 2007 at 11:02 am

    One of the problems leveled against the peer reviewed literature is that it is slow at validation and inefficient. With 70% or so of manuscripts being rejected from most conventional journals papers tend to be seen over and over again wasting researchers time. Time they could be spending actually finding out stuff.

    With PLoS ONE we want to cut down on this inefficiency and waste. The truth is though that some papers are difficult to judge and some are easy. Over-refereeing the easy papers is as wasteful as under-refereeing the difficult is dangerous.

    If a paper is being handled by an editor who is expert in the subject matter why shouldn’t that Editors judgment be trusted, especially if the Editor is identified and that decision open to public scrutiny.

    Most of the published papers in PLoS ONE need more than this minimum level of review but we try to give all papers on PLoS ONE the appropriate level of review for their needs.

  7. dave munger Says:
    August 23rd, 2007 at 11:08 am

    That makes sense, Chris, but I do wonder what safeguards you have in place. What if an editor tends to play favorites and bypasses the process for her buddies? Obviously that’s possible even with traditional peer review, but then at least you can go back and look at the original reviews if it seems like there was abuse.

    How would you suggest defining peer review to include journals like PLoS ONE but still not admit pseudoscience?

  8. Chris Surridge Says:
    August 23rd, 2007 at 5:48 pm

    ‘Two reviewers and the editor’ doesn’t guarantee you anything.

    Whatever your process of assessment the quality of what gets published (however that is measured) depends on the editors that you have. In any system the editors can play favorites and the easiest way for an editor to influence the outcome of review is who they choose as referees.

    I could easily set up a traditional peer reviewed journal of pseudoscience, and make a hansom profit doing so. Indeed there are many already in existence, although I’m not going to name names.

    With PLoS ONE the editors are identified on each paper and if they are behaving poorly then that will be clear for all to see. Also if referees are consulted we ask them to allow us to post their reports. Preferably with their names attached or anonymously if they prefer. And the post publication opportunities are there to point out if errors do happen, and those cases will reflect back on the individual editors.

    In the end though any journal has to be judged on what it publishes. If quality papers are appearing in PLoS ONE, and they are, then that is validation enough for our approach. But that is more to do with the quality and integrity of the editors than the precise details of the system being used.

  9. Christopher Jenkins Says:
    August 23rd, 2007 at 6:56 pm

    What about an impact factor cut-off? I assume that PLoS ONE’s is or will be pretty horrendous, but if its process becomes respected enough among academics that they cite its papers often, that seems like a fair standard.

  10. Chris Surridge Says:
    August 23rd, 2007 at 7:31 pm

    Impact factor is only useful for comparing journals with near identical scope and content. Different subjects cite differently, different types of paper are cited differently. Worse yet the distribution of citations to individual papers isn’t normal but governed by a power law, a few papers get most of the citations, most papers get very few. That means that the mean (which is what IF is)has no power to predict what the citations of individual papers will be.

    PLoS ONE’s Impact Factor will be irrelevant as there is no journal with which to compare it. It will also have no bearing on the individual papers published in PLoS ONE.

    The cult of the Impact Factor is positively dangerous to science. It distorts funding and abdicates responsibility for shaping future research to journal editors. We need to find metrics to judge the worth of individual scientific studies independent of the company they keep.

  11. Dave Munger Says:
    August 24th, 2007 at 5:20 am

    I agree, impact factor really isn’t the way to go here. I’d say we probably just ought to pick a provisional definition of peer review, then allow exceptions for specific cases. To me, PLoS ONE, much as I admire its mission, is a borderline case.

    I’m sure most editors, such as Bjoern Brembs, are actually fulfilling their obligations quite well. But the policy allows some wiggle room that makes me uncomfortable.

    I’d like to know more about the PLoS ONE review process before we decide whether to include them in BPR3’s listing. This doesn’t mean PLoS ONE isn’t a valuable resource, just that it may not meet our definition of “Peer Reviewed.”

  12. ocmpoma » peerless Says:
    August 25th, 2007 at 12:04 am

    [...] icon to indicate peer review), which is in and of itself an interesting concept. For me, though, a couple of posts on peer review were more [...]

  13. Blake Stacey Says:
    August 27th, 2007 at 9:57 am

    Here’s a random thought: what if the icon BPR3 adopted had a few different versions? The basic design could indicate that the blog post was commenting on a paper which survived a plain-vanilla peer review, while a modified version of the icon might have a “PLoS ONE” tag beneath the picture. Another version might say “conference” or “arXiv”.

    In short, the icon should say up front what kind of journal you’re talking about, so you can then get on with discussing the content.

  14. Christina Pikas Says:
    August 27th, 2007 at 10:13 am

    In re conf papers — that really matters in some disciplines. For example, in HCI, CHI is much more difficult to get a paper published in than the journals on the same subject by a major for-profit publisher. Also in engineering, as another commentator said. There are definitely levels of peer review even within the same field between different conferences or meetings — it’s difficult to really make some overarching rule that will make sense.

  15. RPM Says:
    August 27th, 2007 at 11:45 am

    Dave, I can’t believe you’re arguing that PLoS ONE isn’t a peer-reviewed journal. I can understand excluding pre-print servers and conference abstracts, but if you’re going to be picky regarding which peer-reviewed journal fits your mission, you’re going to walk a slippery slope. PNAS, for example, publishes both Track I and Track II articles. Will you exclude Track I PNAS articles (they do not have to be peer reviewed)?

  16. Dave Munger Says:
    August 27th, 2007 at 12:32 pm

    RPM:

    Well, some of the articles in PLoS ONE are peer-reviewed, and some, apparently, aren’t. It strikes me as odd to say that we should slap a “peer-reviewed” icon indiscriminately on both types. Ditto with Track I PNAS articles.

    BPR3 is about identifying peer-reviewed research, not all things that are good in the world.

    Christina:

    I have a couple problems with conference presentations. First, the actual presentation is generally not reviewed–at best it’s an extended abstract. Second, these things are not usually archived. If those two criteria are met, then they also meet my provisional definition of peer review, and so they could be included.

    “Level of difficulty” doesn’t really play into it, in my view. Generally good conference presentations turn into publications anyway, so why not wait until they are published to put the BPR3 icon on them?

  17. Chris Surridge, Managing Editor PLoS ONE Says:
    August 29th, 2007 at 6:52 am

    I hate to throw in another complicating factor here but you can’t assume that even a journal such as Cell or Science or Nature have had two referees and and editor on all their papers especially if you want this to apply to the final version of the paper. If a paper goes through several rounds of revision then referee fatigue often sets in. You might know that a journal aims for their papers to be seen by 2-3 external referees but I can’t think of a journal which actually tells you how many referees were used on each paper (unless the refereeing process is open such as with Atmospheric Chemistry and Physics). And did the two referees have to both agree to publication? With any journal you are basically relying on the authority of the editors to validate the work published. PLoS ONE is no different and the Editor who validated the work is known, something that few journals other than the PLoS Journals do.

    In some ways I’m not too worried as to whether PLoS ONE does or does not fit a particular definition of ‘Peer-Review’. What I am concerned about is that PLoS ONE uses a system of review by scientists that makes it a trustworthy source of information.

  18. Dave Munger Says:
    August 29th, 2007 at 7:07 am

    Those are some excellent points, Chris, and that’s part of why we should definitely keep this discussion open. However, I disagree with your argument that if the final version of a paper isn’t refereed, then it hasn’t undergone peer review. Typically in these situations, reviewers might say something like “if x and y are fixed, then I’d accept the paper.” In those cases, all that’s necessary is for the editor to check whether x and y have been done. That’s different from fast-tracking a paper from the start.

    That said, maybe we should consider definitions of peer review such as your statement “PLoS ONE uses a system of review by scientists that makes it a trustworthy source of information.”

    We’d need to generalize it to cover disciplines outside the sciences. Perhaps something like this would work:

    Peer reviewed work is work that has been systematically reviewed by qualified scholars within the field of the research and published with their approval in an edited scholarly journal, book, online publication, or other form that is considered a trustworthy source of information within that field of study.

  19. Chris Surridge, Managing Editor PLoS ONE Says:
    August 29th, 2007 at 11:05 am

    I completely agree of course that the editors can check to see whether referees comments have been addressed and indeed to decide whether the referees comments are relevant in the first place. The problem is BPR3 needs a solid, easily applied definition of ‘peer-reviewed’ and that isn’t an easy thing to achieve in a way that doesn’t exclude valuable resources like PLoS ONE, Biology Direct, ArXive, Atmospheric Chemistry and Physics etc which are exploring alternative mechanisms of validation. Equally two referees and an editor would include a whole raft of journals with very dubious publishing records. I haven’t got a solution to this except to say that transparency has to be a part of it.

  20. Dave Munger Says:
    August 29th, 2007 at 11:54 am

    Maybe instead of a hard and fast definition, we should offer some guidelines:

    –Reviewed by experts in field
    –Edited
    –Archived
    –Clearly stated publication standards
    –Viewed as trustworthy by experts in field

    Anything else?

  21. Would a community-edited journal count as peer-reviewed? | BPR3 Says:
    September 10th, 2007 at 10:34 am

    [...] responded that in this comment I’ve come up with a set of guidelines that I think are probably pretty close to the final [...]

  22. The INFO Project Blog » Blogging Lessoned Learned: Two Top Ten Lists! Says:
    January 21st, 2008 at 11:39 am

    [...] things in polls and surveys. SurveyMonkey, QuestionPro, Survey Gizmo 2. Creating transparency, minimizing waste in peer review system of publishing. 1. Reaching and pulling information from a critical mass of experts who’ve learned [...]

  23. Iiro P Jaaskelainen Says:
    April 19th, 2008 at 2:24 am

    My own experience with PLoS ONE from some months ago was that the quality of the editor-reviewer comments was much higher than in most other journals, we got those comments a lot faster than in most other top journals (actually faster than it usually takes to get the normal “thanks for submitting but due to general readership…” from Nature). So, even though in our case the manuscript was accepted after a single editor-reviewer and major revisions (that really helped improve the quality), my experience was that the review process was of perhaps highest quality I’ve ever been through.

    I have also noticed that what gets published in PLoS ONE in my own research field are typically papers that are thorough, sound, with very interesting findings. I have not had time to comment those papers on the PLoS ONE site, and I don’t know how well that procedure has taken off in general, but certainly with respect to whether PLoS ONE publications have contributed to science, I’d say definitely as much as any of the multidisciplinary top journals.

    With regards to impact factor, I just for fun looked at how the PLoS ONE papers that appeared in the late 2006 were cited during 2007, and the figure is in fact quite promising. I think this is another indication that PLoS ONE does publish highly relevant findings. So, I’d be quite careful not to classify this journal out from the list of peer-reviewed journals, I think there are many ways to implement the review process, and the end result (quality of published work) should be what counts.

  24. Michael J. Wise Says:
    May 2nd, 2008 at 11:10 pm

    While the quality and timeliness of refereeing is clearly extremely important in shaping attitudes to a journal, the discussion here is missing one important issue: the politics of impact factors. Employment and promotion committees may know nothing about particular research areas, but they do know about impact factors. At a higher level, government policy, e.g. Research Assessment exercises in the UK and Australia, also use them to rank journals and thence the work of those being surveyed. In other words, you may think that impact factors are tosh, but they are a political reality.

  25. Vikram Saini Says:
    May 20th, 2008 at 7:11 am

    We have published two papers in PLos One by now and very frankly, the process of review has been very stringent and of extremely high standards. I wont be overstating if I say it had been better than many reputed journals.The very debate whether its peer reviewed or not looks vague to me. There is no doubt what so ever in our mind regarding that. The wide acclamation that we received from pioneers across the globe in a relatively quick time further speaks for the credibility that PLos ONe has generate in a very short span of time.

  26. Patrick Sullivan Says:
    June 2nd, 2008 at 7:53 pm

    My experience is the same with PLoS ONE — the peer review was certainly not lacking. One manuscript that PLoS ONE rejected I submitted to a “traditional” journal with an IF of 4 that accepted it with minor revisions; the other ms I submitted there got very thorough reviews that were helpful. THe idea that “when money is involved other considerations” are in play doesn’t hold water, because many well-respected print journals have page charges that add up to nearly as much — perhaps more if one buys reprints so as not to run afoul of copyright restrictions for distribution.

  27. luke Says:
    July 3rd, 2008 at 2:19 am

    PLos One’s mandate seems to be the publication of technically rigorous research regardless of importance.

    But I might argue that what constitutes acceptable rigour should depend very much on the importance of the subject matter and the ease of gathering appropriate data. Is a cross-sectional study using exclusively self-report data adequately rigorous? If it’s addressing a peripheral question in some long-established field of study where better data are available, probably not. But if it tells us something important, central and useful in a developing field of study, maybe it is meritous of publication, warts and all.

    In other words, I suspect it’s an illusion that we can separate the evaluation of rigour from the evaluation of importance. They’re joined at the hip.

  28. luke Says:
    July 3rd, 2008 at 2:21 am

    (Where ‘meritous’ = meritorious)
    :)

  29. KFIR OVED Says:
    July 4th, 2008 at 9:57 am

    Hi scientists,

    “Peer review” like its name is peer review. The high impact journals do not give you a real chance in case you or your mentor are not a well known. The populism in these journals makes me sick and lately I find less and less good and original papers in them. The “not in the focus of the general reader” claim have become a way to reject many papers that are not provocative enough or bring massages that are not dogmatic. In this sense I think Plos One will be better for us the scientists (in particular the young and flexible) to judge the originality and impact of a paper. I believe this is the real peer review…

  30. thinkevolution.net » Blog Archive » What’s the deal with PLoS One? Says:
    July 9th, 2008 at 6:35 pm

    [...] which solicit papers, or journals that accept communicated work, and maybe that system is a little bit broken anyway. Secondly, there is the whole “no paper too small” policy at PLoS One. As the [...]

  31. Antonio Brunetti, MD, PhD Says:
    January 26th, 2009 at 3:53 pm

    In any case, does someone knowswhether or not an IF will be available for PLoS One in future?

  32. predict Plosone impact factor Says:
    March 16th, 2009 at 7:25 am

    Who can predict the impact factor in 2008 for the Journal of Plosone, which has published so many articles.

  33. PLoS One impact factor Says:
    April 11th, 2009 at 9:57 am

    The impact factor of PLoS One in 2008 has been predicted according to the citation in 2008 of its articles pulished in 2006 and 2007. It is between 5 and 6. Well done for PLoS One.

  34. Jasen Wise Says:
    May 29th, 2009 at 9:04 pm

    PLoS One breaks the habit of good science only being published if it has lots of money behind it, and sells the products advertised in the issue. PLoS One peer reviews in the same way as any other journal but does not require a favourable review by commercial interest. Large journals such as Nature frequently publish papers that site PLoS One. That in its self is proof enough.

  35. Andreas Bryhn Says:
    June 10th, 2009 at 4:20 am

    I am surprised about this post on PLoS ONE:s reviewing standards.

    My experience after having published one paper in the journal is that the reviewing process was stricter than for most other scientific journals; my article received comments and requests from the editor and three peer-reviewers, which is one reviever more than the usual standard in my field.

  36. Krishnasastry Says:
    June 17th, 2009 at 3:07 am

    I am surprised at the doubts expressed by many regarding the peer review of PLoS ONE.

    Anybody saying that this journal does not ‘peer-reivew’ is rubbish. I have published a paper in this journal recently, i had 2 and half (A4 size, single space) comments by two reviewers. One has gone figure by fig, line by line and asked some very, very pertinent questions on the concept we are working. The second had asked questions (very important) regarding the hypothesis. He also suggested us to do additional experiments. We have done the same and found his comments very informative. I suppose the editor also read our paper before coming to his conclusion.

    Now, how can people comment that PLoS ONE does not have a rigourous peer-review? I consider the review much better than some the journals in IF 4-6 range in which i published earlier! I received comments from a JBC reviewer that were hilarious! We said one protein activates a phosphatase and the reviewer says that i have not shown any data about the phosphatase activity of my protein! Hence, this paper should be rejected!

    Finally, Impact factor is not a substitution for reading the paper.

  37. Samuel Says:
    June 25th, 2009 at 11:30 am

    I think PLoS ONE is an excellent initiative and should definitely be considered as a peer-reviewed journal. My only concern is that it is not indexed in Web of Science. As a young scientist, this makes a difference since most people, when they want to assess your “quality”, the first thing they do is to type your name in WOS and largely judge you based on the results (number of papers, citations, h-index,…). I know it’s not a good method and it’s a bad mirror of one’s scientific quality, but it’s an easy and widespread way to proceed. So, no IF doesn’t bother me, but having my papers indexed in WOS with how many times they are cited is important. Why isn’t it possible, since you can find the PLoS ONE papers in Web of Knowledge?

Leave a Reply

Icons by N.Design Studio. Based on a theme by Ben Swift.
Entries RSS Comments RSS Log in