How do I referee a paper? Updated below

We all know the critical importance of peer-review.  It is the main form of quality control and feedback on research.  However, to an early-stage researcher (like me), it can sometimes be a confusing system/process.

Accordingly, there are several treatises on the scientific refereeing process that give guidance.  Two (very different) examples from computer science -- this 1994 article by Parberry and a more recent one by Cormode -- offer great advice (though the latter might be a shade mischievous).

Here, I'd like to solicit broader advice from the more experienced members of this community about the review process, with particular regard to the peculiarities of theoretical computer science.


What are the main criteria for determining the significance of a paper's results?  How do I judge whether a paper should be accepted to the conference/journal? Is it important to verify correctness?
What are the main elements of a referee report, and which parts are most important? Is it always necessary to give a recommendation of (non)acceptance? What goes in the report and what goes solely to the editor?
How does assessment for conferences differ from that in journals?  How do reports for conferences differ from those in journals? (How on earth do I rate my "confidence" in my recommendation?) Should the journal version be significantly different from the conference paper?
What if I don't understand the paper? ...the proof?  (Is it my fault or theirs?)
What about typographical/grammatical mistakes? What if there are a lot of them?
How much time should I spend on a report?
How many reports a year am I expected to write? When is it acceptable to refuse a request to referee?


Of course, any other relevant questions and answers on this topic are encouraged, since this is CW.

This question is inspired by (stolen from) a similar post at MathOverflow.

Update 15/02/2011:

I am still very interested in getting more input on this question especially with regard to reviewing conference papers and program committee membership. (These two roles are themselves different beasts, and both very unlike being a referee for a journal article, IMO.)  Granted, program committee membership is rarer than refereeing or reviewing (and it hasn't been my privilege yet), but is a responsibility that every researcher in theoretical computer science must take on eventually.


Time.  How much time am I expected to spend as a committee member or conference reviewer?  Given the probability that I could get ten or perhaps many more to handle in the space of a few weeks, how do I avoid running out of time?  What are the most important things to spend time on?
Confidence.  What if the paper is too far from my area of expertise?  What factors should go into nominating/asking someone else to review a submission?  If it is not too far from my area of expertise and I elect to review it, when is it permissible to give a confidence rating of 1?
Criteria.  There are critical differences between journals and conferences.  Some very important papers are not published in journals.  Some very important papers did not previously appear in conferences.  What are the most significant distinctions in criteria on which to assess papers in these settings?
Recommendations.  Inherently, there are fewer recommendations that can be offered to the authors of a conference paper, primarily due to space and time constraints.  Also, there is usually only one round of review.  Another consideration is that my report becomes public to the entire strong committee.  What is the scope of suggestions/directives that I can offer?


As before, if you think I've missed out on asking any particular questions, do let me know, or edit directly.  This is CW, after all.

These new thoughts are partly motivated by reading a paper that Suresh mentioned on his blog.
To the best of your knowledge, does the paper make a significant, well-presented, and correct contribution to the state of the art?  If the paper fails any of the three criteria, it's fair to reject it for that reason alone, regardless of the other two.
Here's what I think a report should contain.  Everything should be visible to the author, except possibly for serious accusations of misconduct.

a. A quick summary of the paper, to help the editor judge the quality of the results, and to help convince both the author and the editor that you actually read and understood the paper.  Place the result in its larger context.  Include a history of prior versions, even if the authors include it in the submission.  Be respectful, but brutally honest.

b. A discussion of the strengths and weaknesses of the paper, in terms of correctness, novelty, clarity, importance, generality, potential impact, elegance, technical depth, robustness, etc.  If you suspect unethical behavior (plagiarism, parallel submission, cooked data), describe your suspicions.  Be respectful, but brutally honest.

c. A recommendation to the editor for further action — accept, accept with minor revision, ask for a second round of reviewing, or reject outright.  Keep in mind that you are making a recommendation, not a decision; if you can't make up your mind, just say so.  Be respectful, but brutally honest.

d. More detailed feedback to the author — more detailed justification for your recommendation, requests for clarification in the final version, missing references, bugs in the proofs, simplifications, generalizations, typos, etc.  Be respectful, but brutally honest.
Conference reports should be shorter; program committees have hundreds of papers to consider at once.  Whether there should be a difference between conference and journal papers is up to the journal (and indirectly, up to the community).  Most theoretical computer science journals do not insist on a significant difference; it is quite common for the conference and journal versions of a theory paper to be essentially identical.  When in doubt, ask the editor!
If you still don't understand the paper after making a good-faith effort, it's the author's fault, or possibly the editor's, but certainly not yours.  The author's primary responsibility is to effectively communicate their result to their audience, and a good editor will send you a paper to referee only if they think you're a good representative of the paper's intended audience.  But you do have to make a good-faith effort; do not expect to immediately understand everything (anything?) immediately on your first reading.
If there are a lot of errors, don't even read the paper; just recommend rejection on the grounds that the paper is not professionally written.  Otherwise, if you really want to be thorough, include a representative list of grammar, spelling, and punctuation mistakes, but don't knock yourself out finding every last bug.  Be respectful, but brutally honest.
Expect to spend about an hour per page, mostly on internalizing the paper's results and techniques.  Be pleasantly surprised when it doesn't actually take that long.  (If it takes significantly less time than that, either the paper is either exceedingly elegant and well-written, you know the area extremely well, or the paper is technically shallow.  Don't confuse these three possibilities.) 
You should write at least as many referee reports as other people write for you.  If this takes more time than writing your own papers, you're not spending enough time on your own papers.