ResearchBlogging.org and PLoS work together to measure the impact of journal articles

December 17th, 2009 News 7 Comments
By Dave Munger

ResearchBlogging.orgWith over 800,000 journal articles published in 2008 alone, it’s impossible even for experts to read all the peer-reviewed research published in their fields. So how do they choose which articles to read? How do non-experts decide which articles are the most important? Until recently, there really wasn’t an effective way to assess the importance of a given journal article. While Thomson Reuters’ Impact Factor purports to measure the impact of an entire journal, this measure alone can’t say how important a particular article published in a journal is. Most of the articles published even in the most well-regarded journals don’t have especially dramatic impacts.

A new movement is afoot to improve on this system, and ResearchBlogging.org is proud to be a part of it. Instead of assigning ratings to entire journals, “article-level metrics” strives to assess the importance of each individual article published. The highly respected journal publisher PLoS began its article-level metrics initiative with several basic tools in March of 2009. But as PLoS ONE publisher Peter Binfield explained in a recent presentation, their plan included rolling out several more advanced features over the course of the year. Today marks the launch of one such tool, a partnership with ResearchBlogging.org to identify—nearly in real time—well-considered blog posts written about their articles.

ResearchBlogging.org now has over 900 registered bloggers who discuss peer-reviewed research in five different languages. Visitors can find blog posts about research by searching for topics that interest them, or by searching for a particular journal article. The relationship with PLoS allows people reading the original articles on the PLoS site to link directly to blog posts about the article.

Every PLoS article has a “metrics” tab offering several different measures designed to help readers assess its impact. Readers can see how many times the article has been viewed, comments made directly on the article, and a user-generated star-rating. Now for the first time they will also see links to posts in blogs registered and approved by ResearchBlogging.org. Since the site carefully vets every blog and rejects both blogs and individual posts that don’t meet its guidelines, these posts represent the best commentary the internet has to offer about peer-reviewed research, often from experts in the field. This offers an important early means of assessing the impact of an article. While PLoS also counts the number of citations of an article by other peer-reviewed articles, these typically take a minimum of several months to get published, while blog posts about articles often appear within a week of publication—and even on the date of publication, since PLoS offers bloggers the opportunity to see pre-release versions of its articles.

Blogs offer another benefit over some other article-level metrics. As Cameron Neylon and Shirley Wu noted in an article in PLoS Biology, experts may be reluctant to offer comments directly on journal articles, because they often don’t receive academic credit for their remarks. Blog posts on ResearchBlogging.org are eligible to be selected as the Blog Pick of the Month at PLoS ONE, and notable posts are recognized daily with Editor’s Selections on ResearchBlogging.org—all accomplishments that can look impressive on an academic curriculum vitae. ResearchBlogging.org now indexes over 8,000 blog posts, including over 700 citations of PLoS articles, so these incentives appear to be working.

For more information about this new way to assess the impact of journal articles, check out this one-minute demonstration on the PLoS website. There is also a full explanation of article-level metrics on PLoS here.

To register your blog with ResearchBlogging.org and participate in this program, click here.

Neylon, C., & Wu, S. (2009). Article-Level Metrics and the Evolution of Scientific Impact PLoS Biology, 7 (11) DOI: 10.1371/journal.pbio.1000242

7 Responses to “ResearchBlogging.org and PLoS work together to measure the impact of journal articles”

  1. Alejandro Montenegro-Montero Says:
    December 17th, 2009 at 3:53 pm

    Also, molecular biology-related articles can be highlighted in our “Picks of the week” at MolBio Research Highlights :-)

  2. David Colquhoun Says:
    December 17th, 2009 at 3:54 pm

    The idea that metrics, at any level, can subsitute for reading and understanding is simply ludicrous. It corrupts science itself and it is very sad that PLOS should encourage the idea.

    One problem, it seems to me, is that ‘bibliometrics’ has become a job in its own right. The one conclusion that a bibliometrist will never come to is that his subject is useless and corrupting. One might as well ask a crystal ball-gazer to condemn fortune-telling.

    Perhaps the best way to solve the financial crisis is to fire the ever-increasing army of people who write about science in the hope that it might leave a bit of money for those who want to do science.

  3. Dave Munger Says:
    December 18th, 2009 at 2:10 pm

    David,

    I think you’re defining “metrics” a little too narrowly. No one is suggesting that metrics will substitute for reading and understanding.

    And you can fill in any occupation you’d like in this sentence: “The one conclusion that a ________ will never come to is that his subject is useless and corrupting.”

    If you’d like to criticize bibliometrics, you might consider offering actual evidence, rather than analogies or truisms.

  4. Mr. Gunn Says:
    December 18th, 2009 at 3:32 pm

    David – think of metrics not as an assessment tool, but as a discovery tool. Whether you think the paper is a good one or not, there’s a good chance that you’ll be interested in reading it if your colleagues are reading it. Think of it like Netflix ratings and Amazon related items. It’s a way to get a related items feature that’s more sophisticated than keyword matching, yet without having a full-blown semantic structure in place.

    As a representative of Mendeley, I can say that they’re also working with PLoS and other document repositories such as publishers and libraries to make sophisticated article-level metrics available in near real-time. The recent structure paper retraction scandal might well have been headed off if the review cycle operated publicly without letting things fester in private.

  5. Sarah Says:
    December 25th, 2009 at 8:45 am

    Hi Dave – sounds great. Are there any plans to work together with publishers in other disciplines, or with Arxiv?

  6. Dave Munger Says:
    January 5th, 2010 at 5:02 pm

    Sarah: Indeed, there are. That’s one of our primary goals. Of course, it’s a bit of a chicken-egg thing. They need to see more posts on ResearchBlogging.org in order to commit to working with us, and we’d get more bloggers interested if more publishers were involved.

  7. David Colquhoun Says:
    January 16th, 2010 at 6:52 pm

    Dave Mungar says

    “I think you’re defining “metrics” a little too narrowly. No one is suggesting that metrics will substitute for reading and understanding.”

    I disagree, Promotions committees and even funding agencies are doing precisely that. The problem is perhaps less serious in better institutions, where people prefer to think for themselves, but even their the corruption of metrics is creeping in.

    You say also
    “And you can fill in any occupation you’d like in this sentence: “The one conclusion that a ________ will never come to is that his subject is useless and corrupting.”

    True, but try filling in my occupation, science. What do you think?

    And you end by asking for evidence! You are on very thin ice their. My job is based entirely on being abile to produce convincing evidence for causal relationships. Bibliometrics, in my view, is not.

    Mr Gunn says
    “David – think of metrics not as an assessment tool, but as a discovery tool.”

    I don’t think that any self-respecting scientist would consider a populariy poll was a sensible way to decide whether or not to read a paper. I don’t understand you remark about how your ideas would have helped with retraction of fraudelent papers. I’d guess that such papers are very popular (until the problems are discovered).

Leave a Reply

Icons by N.Design Studio. Based on a theme by Ben Swift.
Entries RSS Comments RSS Log in