The first step in reviewing a paper is to get an understanding of the venue to which the paper has been submitted. Is it a conference or a journal? What are the expectations of the conference or journal? What is the acceptance rate? What types of papers are typically accepted? Next, you must determine if you are an appropriate reviewer for the paper. Is the paper close to your area of study? Do you know literature relevant to the subject of the paper? Do you have the time to complete the review as requested, including any background study you must do to complete the review effectively? (Of course, if I have asked you to do the review, it is safe to assume the answers here are all "yes" :-)

In general, journals have higher expectations for the thoroughness of reviews than do conferences, but that is not always the case. In either case, you have two primary responsibilities: determine if the paper is acceptable for publication in the selected venue and provide constructive feedback to the authors to improve the paper.

It is important to direct your efforts toward improving the paper. Improved papers help the entire community. Focus on constructive comments. If there are weaknesses in the paper, try to focus on how to address them and strengthen the paper.

Outline of a Review

The general outline of a review should include:

• A summary of the paper in your words
• A brief statement of what you think the author is claiming are the contributions of the paper
• A detailed analysis of the paper's organization, presentation, technical accuracy, and results
• Answer the question, "Did the authors adequately support their claims?"
• Review the adequacy of the authors' citation list.

The summary of the paper is important because you want to show the authors how well you understood their material. This provides feedback to the authors on how well they explained their work. It also provides context for your comments on the paper. For a conference paper, the summary should be a good paragraph or so. Put some effort into describing what you thought is the message the authors are trying to deliver. For a journal paper, this summary should be half a page to a page of text. Don't just restate the abstract....put some real effort into summarizing your own understanding.

Identifying the contributions of the paper is also important. Do the authors state this explicitly? You must try to figure out what the authors feel are the contributions, even if you think the contributions are different.

Many conferences and journals provide a list of things to consider, such as (from ICRA):

• Originality of Concepts
• Relevance of Applications
• Technical Soundness
• Thoroughness of Results
• Clarity of Presentation

The analysis of the paper is the most important part of your review. It is your responsibility to do whatever you need to do to complete a thorough analysis. This often means you are expected to review the authors' prior publications and any key publications the authors have identified that their work is based on. Again, the expectations for this "background check" are generally higher for journals than for conferences. This section should be at least several paragraphs for a conference publication and often several pages for a journal publication. Typographical corrections are helpful for a conference paper, but less valuable for journals as editors and proofreaders will take care of that.

Putting contributions in the context of prior publications is an important part of academic papers. Do the authors provide an adequate summary of relevant prior work? Have the authors missed important prior publications? Do they accurately describe the prior work? Do they point out how their work is different?

If there are specific publications the authors have missed, include a complete citation. If their refernces are incomplete or incorrect, point this out. If they have incorrectly summarized the importance of a paper, provide your interpretation.

Confidential Comments to Editorial Committee

However, most review processes provide an opportunity for the reviewer to make confidential comments that the authors will not see. It is not necessary to make confidential comments and, in fact, confidential comments should be rare. It is not necessary to repeat anything that you are telling the authors.

---

rvoyles [at] purdue [dot] edu