Writing Good Reviews

Q: “How do I write a good review for a journal? I don’t want to use the reviews I have received as an author because many are completely unstructured. ”Portrait: John Robert 'Rob' Warren

Author: Rob Warren, University of Minnesota

Current Editor of Sociology of Education

 

Everybody hates bad article reviews.  Editors, authors, even the other reviewers.  In this post I describe what I think makes a good article review, and I offer a few practical pointers for beginners.

Remember the manifest purpose.  There are lots of good reasons to agree to review an article—to find out about state of the art research; to contribute to the collective good;  to help shape the area, etc.—but don’t lose sight of the fact that what you are mainly doing is making a recommendation to the Editor.  Your review is, first and foremost, a letter to the Editor in which you make a recommendation about what decision your think she or he should make.

Whatever else you put in your review, be sure that you are very clear in justifying your recommendation.  And, make sure that the specific recommendation you make to the Editor (i.e., accept, reject, invite a revision and re-submission) matches the tone and content of the review.  Imagine a movie review that gives a new film 2 out of 5 stars—but does not explain what’s wrong with the film.  Or, imagine a movie review that gives a new film 5 out of 5 stars but then goes on to trash the plot, the script, the acting, and the special effects.  Neither movie review would be very helpful.  The Editor has to make a decision based on your recommendation.  Telling the Editor to “accept” the paper and then writing two pages about flaws in the research design puts the Editor in a tough spot.  So does recommending that the Editor “reject” the paper when your review contains little by way of critique.

Remember the “Golden Rule.” The manifest purpose of an article review is to provide a recommendation to the Editor about what to decide about a manuscript.  But we all know that the peer review process can be a major avenue for improving our research.  Reviewers have read things we haven’t read, they have theoretical perspectives that may be useful to us, they know about data or measures we don’t already know about, they are familiar with new methods we haven’t heard of, and they may interpret our results in surprising and constructive new ways.   As an article reviewer, you can make a huge difference in helping someone to improve their research—and thus the state of knowledge in the field.

As an author, you want reviewers to provide lots of thoughtful, constructive feedback based on a close and engaged reading of your manuscript.  Don’t be a free rider!  Be the reviewer you wish would read your next submission.  Take the time to be helpful.  Mention the new book or article that the author might not know about (and provide the citation).  Carefully explain why the author’s methodology is in need of improvement.  Engage in the theoretical discussion.  Challenge the author to think things through more carefully.  Etc.  Show the author the respect you would like to receive.

Timeliness is better than godliness.  A Pulitzer Prize-worthy review is useless if you send it in weeks (or months) after the journal’s deadline.  When you are invited to review an article, the invitation will usually say specifically how quickly the Editor hopes to receive your evaluation.  If you know you cannot stick to that time frame, then decline the invitation to review—and provide several possible alternate reviewers.  If you agree to review the manuscript, do it by the deadline.  Again, be the reviewer you wish would read your next manuscript.  Return reviews as quickly as you wish everyone else would.

Structure your review strategically.  A long, meandering, unstructured review is not very helpful.  Be clear, concise, and to the point.  Emphasize points by putting them first.  Here is how I wish every review were structured:

  1. Start by summarize the author’s research question, theoretical perspective, data, research design, and major findings.  Do it in just a few sentences, and don’t make any evaluative statements.  This tells the author you read the paper and understood what they were trying to say. (Length: 1 short paragraph)
  2. Provide an overview of your evaluation of the manuscript.  Many Editors will say not to include your specific recommendation, but others disagree.  Say, overall, what you think of the research and the manuscript.  (Length: 1 very short paragraph)
  3. Say what you like about the manuscript.  This is helpful to the Editor, but it also cushions the blow for the author since everything after this section will be more negative.  (Length: 1 short paragraph)
  4. Describe a few major weakness or limitations—and in declining order of importance.  If there is fatal flaw, lead with that.  Don’t list problems in the order they occurred to you.  List them in the order of how serious they are. (Length: Depends.  Maybe a page?)
  5. Describe a few less serious problems or issues.  These are things that are not “make or break” issues, but that you’d like to mention by way of being helpful and constructive.  (Length: Depends.  Maybe a long paragraph?)

If you add it all up, that comes to about two pages.  However, the Editor and author will know after less than a page whether (a) you read/understood the article; (b) what you basically think of it; and (c) what you see as its major strengths and weaknesses.

Don’t pretend to be expert on everything.  If you know a lot about the substantive area or empirical topic, but nothing about the methodology, that’s OK.  Likewise, if you know a lot about the methodology but nothing about the theoretical framework or empirical subject, that’s OK.  Comment on those areas about which you are expert.  Trust that the Editor will enlist other reviewers to comment on other aspects of the manuscript.

It’s not about you.   Evaluate the paper in front of you, not the one you wish was in front of you—or the one you would like to write.

Don’t overdo it.  Unless there is some exceptional circumstance, I do not think people should spend more than about 4 hours on an article review.  I recommend taking two hours to read the manuscript and take notes, and then another two hours to formulate and draft the review.  Or less. If you let it take longer, you will resent it—and thus be more likely to decline to review the next manuscript sent to you.  On average, you should expect to review three times as many manuscripts as you submit.  Make sure you review manuscripts well, but in a sustainable way.

They have people for that.  You do not need to comment on grammar, spelling, or smaller matters of style (unless the manuscript is so badly written that you feel compelled to make it a core issue).  Even in this low budget era, journals usually have copy editors.   You also do not need to tell authors how to format their references, tables, or figures.  Most journals contract with someone to do those things.  Spend your precious time and energy on the central task at hand: Making a recommendation to the Editor and providing constructive feedback that will help the author improve their research.

Advertisements

3 comments

  1. What a fabulous post. This is so helpful! Thanks to Rob for taking the time to share his thoughts on this issue.

  2. I second Amy’s comments. I’ll definitely pass this along to my class to let them know both what to expect from reviewers and how to write their own reviews.

    Two questions, though:

    1.) Any thoughts on compiling a “chronological” list comments questions by page/line number? A number of the reviews I’ve received have done this, and I usually find it helpful. I sometimes do it in my own reviews, but can’t decide if it’s always helpful to the author(s) and editor(s) or if it just makes me more nitpicky and my thoughts more scattered.

    2.) Other than the obligatory saying something positive before saying something negative, is there anything people can do to foster more productive reactions from authors? It seems like an awful lot of our jobs come down to how people give and receive evaluations.

    1. Hi Corey,

      1. My preference is not to list comments chronologically because it doesn’t always make it very clear which comments are most important or which issue is most in need of attention. Perhaps the “minor” comments at the end can be listed in page number order, though.

      2. Nobody liked bad news, and nobody likes to hear that the article they spent so much time on is flawed. Tell it like it is. If the author doesn’t respond productively, I’m not sure there’s much you (as a reviewer) can do!

      Rob

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s