Michael's Happenings

Notes from Professor Carol Saunders, "Gatekeepers or Diamond Cutters: The Role of the Reviewer", Jul 26

Carol is the Editor-in-Chief of the MIS Quarterly, a top-three IT academic journal. In 2006, they had 5 issues (4 quarterly and 1 special), and of 346 submissions, there were only 43 papers published. There are 10 senior editors, and 40 associate editors. The average turnaround time is 120 days or less (although the second round can take longer). Carol said that she rejects 50-60% of submissions outright, and these rejects all go out with a covering note as to say why. If the paper goes out to an associate editor (AE) and it is rejected then, they must write a reject letter to explain why, what the authors could do going forward, etc. When the AEs have finished their reports (for accept or reject), the reports go back to the senior editor for a final decision to be made.

Carol has been the Editor-in-Chief since January 2005, and will end her role in December 2007. When she joined, her vision was to:

  • Increase the types of papers published
  • Increase the number of papers published
  • Maintain the quality

When she joined, her strategy for accomplishing this was to broaden the reviewer base (eg, non-North American reviewers, and additional topic areas covered), and more “diamond cutters” (that is, how to take a paper on the margin and increase its quality).

Advantages of Reviewing
Becoming a reviewer brings numerous benefits to the reviewer:
1. It is important for your discipline. The input of reviewers keeps the papers honest, and keeps the quality up. It means that when subsequent readers go through a paper, they know that it has already been signed off by some others.
2. Helps with your writing. Eg, you can see how others have done well and where their writing is confusing
3. Helps you stay current and gain insights.
4. Is the professional thing to do.
5. Allows you to compare your assessment with others. Eg, ongoing peer comparisons because you often get to see what other reviewers have said.
6. Builds positive reputation and social network of grateful editors and colleagues … assuming (a) you do a good job, and (b) keep to time deadlines.
7. Is first step to an editorial board invitation, eg,
8. (did I say this already!) is important for your discipline!

Not everyone in a discipline does reviews:
– they are asked for a review by (a) it’s not their area of expertise, (b) you’ve already rejected the paper for another journal, (c) bad timing due to semester / teaching load, and (d) academics who aren’t good citizens (they submit papers but don’t review papers for others).
– they are not asked to review. That is, their expertise isn’t known or public.

“Why are we all so cranky as reviewers?”
Why is this?
1. Reviewing is supposed to be critical … we have to define what’s right and what’s wrong, however we often focus almost exclusively on what’s wrong. The problems come to the top, and we forget to focus on the good things.
2. There are few extrinsic rewards for reviewing. Eg, reviewing doesn’t help with pay scales, and there is no weight from reviews in performance evaluation.
3. Good reviewing is time-consuming … Carol says it can take at least a day to do it right, and more if you have to check references or methodology.

Evaluation of Reviewers
We are doing well … reviewers find the problems, and few flawed articles get published (in MISQ). At a basic level, we’re doing okay.
However, in terms of extending our capability, we don’t allow good ideas that are phrased wrong or stated poorly to get through. And thus the discipline misses out.

David Harrison, author of Obligations and Obfuscations in the Review Process (PDF) (Academy of Management Journal), advocates a Bill of Rights for Manuscript Editors:

  1. Manuscript authors have a right to respectful and courteous interpersonal treatment … always.
  2. Manuscript authors have a right to full and careful readings of their submissions. Eg, you should be able to sit down and read it in one sitting. Read it at least twice. Don’t review the paper when you’re cranky.
  3. Manuscript authors have a right to expect criticisms of their work to follow the same standards of logic and evidence applied to themselves. Eg, detailed support, findable citations, avoid the tyranny of your own pet ideas.
  4. Manuscript authors have a right to expect criticisms of their work will be prioritized. Eg, fatal flaws first, split and prioritize points, etc.
  5. Manuscript authors have a right to get feedback about their work in a reasonable span of time. Eg, meet review deadlines, like a “speedy trial”. However, you don’t want it to be “too fast” — a good review takes time. Be aware of holidays and academic rhythms (eg, in IS discipline, you won’t get a speedy review in December … due to the annual ICIS conference in the second week of December).
  6. (Carol’s addendum) Manuscript authors have a right to expect that their completed manuscript will say what they think is important for them to say.
  7. (Carol’s addendum) Manuscript authors have a right to expect that reviewers will respect their right to write things that do not agree with what the reviewer has written.

Remember, “a journal is known for the articles it publishes rather than those it rejects!” (Blake Ives, previously Editor-in-Chief, MISQ).

Doing a Good Job as a Diamond Cutter
1. Remember … no article is PERFECT! … even after publication. Can a reviewer overlook some of the problems because there is some good therein.
2. Don’t expect … all prior research to be fully incorporated, a model with all of the constructs, an empirical study with no limitations, the most appropriate analytical technique, etc.
3. Argue the strengths as well as the weaknesses of the paper. Provide (a) useful references … with findable citations, (b) new analysis opportunities, (c) alternative theoretical base, (d) different explanation for results, (e) a cleaner focus, (f) further extensions, (g) implications for practice, (h) suggestions for rewriting that can reposition the work to avoid fatal flaws … if you can do this, you are a master reviewer.
4. For every major criticism, find a suggestion on how to address it.
5. Have a change of mindset … instead of looking for what’s wrong, ask “how can I get this paper published?”
6. Some papers just aren’t publishable, but future research from the same authors could be. Give the authors direction on how to do a better job in the future.

Allen Lee added some additional points (from May 1995):
7. Start out by writing a manuscript summary … it will help the author know whether you “got it” or not
8. Describe and define your expertise
9. Be really specific in feedback. Eg, quote from the author’s work, give page numbers, etc.
10. Comment on tables, figures and diagrams
11. Date your review (unless you’re very late in doing your review!)

Lots of reasons why this is important:
– it helps the discipline grow and address stimulating issues
– develops important streams of research
– gives the reviewers intrinsic rewards

Where can we find more “diamond cutters”?
– register on ISWorld (www.isfacdir.org)
– encourage more participation, eg, changing the reward system/norms. Some journals pay a nominal amount for getting their review in on time.
– train PhD students to review (hey, I’ve reviewed a paper already for the IEEE Transactions on Professional Communications!)
– train our reviewers
– institute author’s ability to respond in the review process
– recognize reviewers when the article is published. That is, when an article is published, the names of the reviewers are listed alongside the article, if the reviewer wants it.
– institute a Developmental AE award

Concluding remark, “may your papers be reviewed in the same way that you reviewed this paper.” (an academic version of the “do unto others as you want them to do unto you”)

Open Discussion and Questions
1. The rejection rate is constant for MISQ, or perhaps even rising. It’s within a bound of 10-15%. How does this correlate with the submission rate? Eg, you can get around some of this by publishing more issues.
At MISQ, we’re publishing more than we used to … but we’re also getting more submissions. Carol said that one reason for a high submission rate is that some people treat the MISQ reviewers as a free review. That is, they can send their paper in for a “free review” that is very thorough. The author can then tidy it up and submit it to a less stringent journal.

Carol said that in terms of “journal slots to fill”, she said that they could publish more … but generally don’t have a sufficient number of good articles to fill what’s available.

2. Does Carol ever give feedback to reviewers on how they’ve done as a reviewer on a paper?
I don’t do it as much as I should … nor as explicitly as I should. She does it sometimes, eg, when the feedback is right on but the tone is too harsh. If someone doesn’t do a good job as a reviewer, then Carol will often not ask them again.

What Carol hopes is that when the post-review packet is sent out to all reviewers, they will compare their feedback to the other reviewers and learn how they could have done better.

3. For the new research, when we finish our research, go through a conference, and get it ready for submission to a journal, keeping in mind that “no paper is perfect”, when do we send it off?
Get it right according to your own requirements, but don’t leave any obvious flaws. If you know that things still need to be worked on, work on them.

Professor Rajiv Kohli is also in the room, and he adds: another idea re flaws … is to acknowledge them. Build a personal network of reviewers so others can go through your work before it gets sent out.

4. Do we need 3 reviewers? Isn’t 2 sufficient?
There are many reasons for having three reviewers … it hedges if someone is late, it provides a way of developing PhD students, and it widens the base of feedback.

5. How do you handle conflict review decisions?
There are a variety of ways of dealing with conflicting review decisions, but you have to rely heavily on your reviewers. In the early stages of the review process, it needs to be made very clear to the authors as to what they are supposed to do to make the paper ready for publication.

6. An idea … a way to overcome the hold-out reviewers is to share the review packets earlier in the process. If the reviewer is kept blind to others feedback, it makes it very difficult.

7. How do you deal with the challenge of being able to guess the identity of an author?
Give each person the same level of respect … regardless of who they are, and how long they’ve been in the field.

8. Does a reviewer use different standards depending on where the paper is going to be published?
Carol … I think yes. The point of differentiation is the level of contribution to the field. Eg, the level of academic contribution is expected to be higher in MISQ than in other journals. Other journals have a lower standard in terms of contribution to add.

Rajiv … I have a slightly different perspective … the level of contribution should only come into play at the point of decision re publication. When you are reviewing it, you should have a consistent standard across all reviews.

Markus … review the work, not the author. Deal with the arguments, methods and evidence … not the author.

Rajiv … for conference papers that have a limit of 10 pages, you have to assume that some things will be missed that could / will be included in the full paper. The reviewer either has to assume that it is going to be forthcoming, or you can ask for more detail.

9. What about the idea of not asking the reviewer to make a recommendation on publication, but just to give a review?
Carol … in many cases the reviewer can’t see the full picture.

Rajiv … in terms of MISQ, if the Senior Editor makes the sole decision, it would be too easy for authors to start attacking the Senior Editors.

10. How often do you see self-plagiarism?
Carol … it’s a big problem in the IS discipline. Carol wrote an editorial at MISQ in December 2006 (PDF) addressing this. There are several things you can do … (a) don’t cut and paste (re-write), (b) if you have multiple authors, have different authors write different portions.

Markus … it’s a function of the performance measure system we work under.

11. What happens when you have a large data set, and different variables, and different models?
Carol … by all means use it a second time if you can make an additional contribution, but don’t overdo it. Some people try to use the same data set for 10 times or so.

Rajiv … I use a very large commercial database (350 fields) for his data set. I use it a lot. When he does copy-and-paste, he color-codes the pieces that have been pasted. This means that it is easy to go back to earlier papers.

Concluding Remarks
Thanks to Carol for a stimulating session and discussion.

Categories: Michael's Happenings