225
$\begingroup$

I've refereed at least a dozen papers in my (short) career so far and I still find the process completely baffling. I'm wondering what is actually expected and what people tend to do...

Some things I've wondered about:

1) Should you summarize the main results and or the argument? If so, how much is a good amount. What purpose does this serve?

2) What do you do when the journal requests you to evaluate the quality of the result (or its appropriateness for the journal)? Should one always make a recommendation regarding publication?

3) What to do about papers with major grammatical or syntactical errors, but that are otherwise correct? Does it matter if the author is clearly not a native English speaker?

4) On this note, at point does one correct such mistakes? Ever? If there are fewer than a dozen? Should one be proof-reading the paper?

5) What do you do when you do not understand an argument? Does it matter if it "feels" correct. How long should one spend trying to understand an argument?

6) What to do about papers that have no errors but whose exposition is hard to follow?

$\endgroup$
3
  • 17
    $\begingroup$ I just want to add a note thanking the community for answering this question. The concensus on point (1) is particular illuminating for this young researcher. $\endgroup$ Aug 25, 2010 at 10:36
  • 4
    $\begingroup$ I just wanted to thank everyone for their very useful comments. I really did clarify things for me. $\endgroup$
    – Rbega
    Sep 9, 2010 at 4:03
  • $\begingroup$ I think the question got answered in sufficient detail, vote to close. $\endgroup$
    – user9072
    Dec 25, 2011 at 19:33

11 Answers 11

99
$\begingroup$

I've been spending a fair amount of time editing a journal this year, and it's pretty amazing what different people think of as a "referee report". The first thing you should keep in mind is that the editors will be incredibly appreciative if you look at the paper in detail and send in comments in a timely manner, whatever the comments are. In my mind a good referee report begins with a very short (a few sentences) summary of the result and the argument. It includes an opinion on whether the result and proof are (i) correct (ii) readable (iii) interesting to lots or only a few people; also (iv) a recommendation on whether it is good enough to appear in the given journal, or alternatively/also comparisons to the quality of one or two other recent papers in the field, or just a statement that the referee isn't sure if it is good enough or not, and (v) a non-empty list of specific corrections/suggestions.
Re (3-4) it's perfectly fine to send in a one-line request that the paper be revised so that it is written in correct English. It's not your job to correct grammatical mistakes if there are more than a few. Re (5-6) if an argument is hard to follow, you can just ask for a revision in which the authors explain more. As an editor, it's quite easy (with computers and e-mail being what they are) to request a revision, even after the referee has only read part of the paper.

$\endgroup$
5
  • 28
    $\begingroup$ This is a helpful answer. I might suggest that if, as you say, many of the reports you get are less than helpful, then maybe you could be clearer about what you're asking for when you solicit them. In my experience, most editors aren't at all clear about what they'd like to know about the paper, and it is I think what lies behind the original question above. $\endgroup$
    – JBorger
    Aug 25, 2010 at 2:25
  • 2
    $\begingroup$ All comments are helpful, there is just a lot of variation. If some is senior, then they will have their own style of referee report and I wouldn't feel comfortable asking them to fill something like a form. Anyway most of the editorial work seems to be dealing with papers that people seem to think are good but no one actually wants to read, and so my attention to individualizing request letters often wavers. If the world was full of people like RBega ..... $\endgroup$ Aug 25, 2010 at 12:40
  • 12
    $\begingroup$ You could always write, "If you are used to writing referee reports in a particular way, please feel free to do so; if not, I've attached a template and checklist that you may find helpful." $\endgroup$
    – JBL
    Aug 25, 2010 at 13:03
  • $\begingroup$ As a referee, I've felt free to contact the editors when I wasn't sure what they were expecting. $\endgroup$ Aug 25, 2010 at 19:31
  • 2
    $\begingroup$ (Re 5-6) It may happen that a paper that otherwise falls squarely in your field happens to borrow a technique from a less familiar area. In that case, a revision may not be necessary; just point it out and the editor can call in an expert to consult on that particular point. $\endgroup$ Aug 25, 2010 at 19:37
67
$\begingroup$

I think your question is so important as to deserve multiple answers, even if there is a good deal of overlap among them. (Indeed, overlap indicates that various respondents feel the same way about something, which is important.) So:

1) Should you summarize the main results and or the argument? If so, how much is a good amount? What purpose does this serve?

Absolutely yes. How much is hard to say: one thing I often do is try to put the paper down and write out a summary of the results (including precise statements) and the basic strategy of the proofs. This gives me something very longwinded. Then I try to pare it down until the point where cutting out anything else would be an obvious disservice to the author or to my explanation. It serves several purposes:

a) You need to have a high level grasp of what is being done in the paper. Writing out this summary allows you to gain this grasp (and this is especially important if the authors have done a bad job of indicating the importance of the paper in the introduction; I find that perfectly good papers are often not properly explained in this way) yourself, and

b) You demonstrate to the author and editor that you have actually read, absorbed and digested the paper. (Or you demonstrate that you haven't!) This is important both as part of the decision process and also psychologically for the author: it is really maddening to get a paper rejected by a referee who you think didn't properly read it or understand it.

c) This factual information forms the basis for the opinions you will render later on.

2) What do you do when the journal requests you to evaluate the quality of the result (or its appropriateness for the journal)?

Then you evaluate the quality of the result and its appropriateness for the journal! This is the most important part of the refereeing process, and you should do it even if you aren't explicitly asked to. (It is very unlikely that the additional opinions you provide will be unwelcome. It is much more likely that you just didn't get sufficiently explicit instructions for whatever reason.)

Should one always make a recommendation regarding publication?

Yes. It needn't always be "absolutely yes" or "absolutely no" -- i.e., you can recommend publication conditional on the author's making certain changes (possibly without looking at the paper again, I mean) or you can recommend publication more or less strongly according to certain information that the editor has and you don't (e.g. backlog), but you should definitely make a specific and unambiguous recommendation.

3) What to do about papers with major grammatical or syntactical errors, but that are otherwise correct? Does it matter if the author is clearly not a native English speaker?

This is a tough one. I think if the errors are truly grammatical and/or syntactic and they are not so serious that they create ambiguity in the meaning or interfere with your undertanding of the paper, then this should be regarded as a "copyediting" issue and be clearly placed at a lower level of importance than matters of mathematical correctness or worth.

As an author, it can be annoying to have the recommendation for publication delayed by a referee who wants to make a big fuss about copyediting issues. (Even more annoying are referees who delve into issues of formatting. I have had referees ask me why certain things are italicized and one referee who sent me an entirely new copy of my paper with the font size changed. I felt that was silly.) By coincidence I got a referee report just this evening, and -- while it did point out a matter of content, for which I am quite grateful -- there were a lot of nagging little things, e.g. "Your notation for cyclic group seems odd to me." Okay, thanks for sharing.

Ideally, there will be an actual copyeditor who steps in after the paper is accepted. Some journals do this (mostly AMS and MAA journals, if memory serves).

4) On this note, at what point does one correct such mistakes? Ever? If there are fewer than a dozen? Should one be proof-reading the paper?

That's totally up to the discretion of the referee. You certainly have the right to do it but are not obliged to. I have myself done it sometimes for authors who are both junior and non-native speakers of English, because I think that they probably will both be happy to receive it and learn from it.

5) What do you do when you do not understand an argument?

Well, that's your biggest worry when you're refereeing a paper, isn't it? I can't give a complete answer here. Certainly you should be prepared to put a fair amount of effort into understanding a crucial point in the paper, including asking friends and colleagues about background knowledge, if appropriate. If after a while you still don't get it, you should say so in the report, and in my opinion you should make clear to what extent the author is at fault for the lack of understanding and what you want to be done about it. As an author, I have gotten comments from referees like "I don't understand the proof of Theorem 13." What am I supposed to do with that information? Tell me exactly what you didn't understand, or tell me that the whole argument is hopelessly vague or flawed, or tell me that you were honestly in a little over your head at this point. But tell me something.

Does it matter if it "feels" correct?

Yes, I think it does, but I don't feel comfortable saying more.

How long should one spend trying to understand an argument?

Again, there is no one good answer to this.

6) What to do about papers that have no errors but whose exposition is hard to follow?

I would say that if you can successfully ascertain that there are no errors, the exposition was not that hard to follow! Seriously, you can make as many specific suggestions/demands for expository changes as you like, but if you have actually understood and verified the paper, then rejecting it just because it is badly written is not a decision I would feel comfortable making as a referee. (As an editor, yes.)

$\endgroup$
2
  • 2
    $\begingroup$ I often recommend changes. I figure it's up to the editor to judge if my demands are an unfair burden on the authors. Sometimes nothing happens, and sometimes changes are made that really improve the paper. It's always nice when the latter happens, especially when you've been working hard on the report. $\endgroup$ Aug 25, 2010 at 12:59
  • 10
    $\begingroup$ I haven't rejected a paper because it was badly written, but I have written quick referee reports to the effect of "This paper looks interesting, but I believe the exposition is poor because of X, Y, Z... if the author makes a serious effort to improve this, I would be quite happy to read the paper closely." $\endgroup$ Dec 6, 2010 at 7:24
57
$\begingroup$

I've refereed at least a dozen papers in my (short) career so far and I still find the process completely baffling. I'm wondering what is actually expected and what people tend to do...

I used to think that I had to check a paper's correctness, but now I think that the main point of my report is to help the editors decide whether they should accept or reject the paper. It's the author's job to make sure the paper is correct. This has few practical consequences:

  • I'll agree to referee a paper even if I don't think I'll be able to understand everything in it, but I'll send it back as soon as possible (within a week or two) if I feel that I can't even tell whether it's interesting or not. In this case, I'll suggest some other names.
  • I do summarize the main result, so that the editors know what it's about. I try to do this in half a page. If I'm not able to do this, it's a bad sign. It has happended that I couldn't tell what the author was trying to do.
  • I give a short opinion about acceptance or rejection, it's usually not hard to decide.
  • After that, I include a list of the errors and typos which I found when reading the paper.

Because of the way I do this, I'm usually partial to a good introduction.

Here are a few things which have displeased me (as a referee) about the process:

  • I recommended that the paper be rejected because even though it's close to my subject, I couldn't understand anything, but the editors said that they had sollicited the paper and they felt obliged to accept it.
  • I made a four-page list of errors and suggested corrections, which the author ignored, and after a couple of back-and-forth, the editors asked could they please just accept the paper as it was.
  • The editor was a friend of the author and felt I should not have been so harsh.

I agree with Richard Borcherds' recommandation that one should write assuming that the author will find out the referee's name. But recall that the author will see your report in a quite different light. I once started a report with something to the effect that the paper was very good and useful and bound to become a reference. A few weeks later, I was talking to the author who said the referee had written a "quite negative report".

This is the way I do things, but I have to say that receiving a six-page report where the referee has checked all my computations (including the signs!) is quite wonderful.

I think it would indeed be helpful if each journal was more precise in saying what they expect of the referee. Conversely, as a referee, I like to know the outcome (some journals received my report and that was the last I heard from them). Sending a reprint of the paper once it has appeared is a nice gesture.

My last comment is a reiteration of the fact that a timely report is important and useful. If you can't do it in two or three months, you've got to ask yourself if you are going to do it at all...

$\endgroup$
4
  • 50
    $\begingroup$ Re your first paragraph ("I used to think that I had to check a paper's correctness ... It's the author's job to make sure the paper is correct"), I disagree. The author has an interest in getting the paper published even if it's incorrect, and the referee is the only instance that can catch serious mistakes before it goes to print. I wouldn't go as far as checking signs, but I would never give the go-ahead if I'm unsure about whether the main results are true. $\endgroup$
    – Tilman
    Aug 25, 2010 at 11:23
  • 3
    $\begingroup$ In very many cases, one can be sure that the main result is true without checking the correctness of all the proofs (for a sufficiently colloquial definition of "sure"). $\endgroup$
    – JBL
    Aug 25, 2010 at 12:51
  • 12
    $\begingroup$ "Sending a reprint of the paper once it has appeared is a nice gesture." Agreed! Unfortunately I have never seen this... $\endgroup$
    – DamienC
    Aug 25, 2010 at 21:19
  • 24
    $\begingroup$ "The author has an interest in getting the paper published even if it's incorrect" This is, to say the least, not in the author's LONG-TERM interest. $\endgroup$
    – JSE
    Oct 10, 2010 at 16:09
38
$\begingroup$

There are only 2 rules that really matter when refereeing:
(1) Either referee it within a couple of weeks, or reply immediately saying you do not have time.
(2) Assume that the author will see your report and will find out who wrote it.

$\endgroup$
6
  • 46
    $\begingroup$ While I do get annoyed when my submitted papers sit in limbo for a year or more, I think that asking all refereeing to be done within a couple of weeks is a bit extreme. $\endgroup$ Aug 25, 2010 at 7:02
  • 3
    $\begingroup$ Refereeing within a couple of weeks might not get the job done: the paper might be too long, might require you to study something outside your field... I find it better to ask from the editor for a time frame, and then agree to it and hold yourself to it, or turn it down. $\endgroup$ Aug 25, 2010 at 12:52
  • 12
    $\begingroup$ I ignored (1) once with bad results. The author (now dead) was great and idiosyncratic with an obscure style. I carried the paper around for a few months hoping to understand it. Finally I wrote the editor, saying I wasn't the right choice of referee, and pointing out a counterexample to one main theorem. The editor must have rejected the paper, since the author (god knows how) discovered my involvement and sent letters around denouncing me--we never really reconciled. $\endgroup$ Aug 25, 2010 at 15:50
  • 3
    $\begingroup$ Getting back a positive referee report within two weeks of submission to math journal would be a pleasant shock to me... (Maybe things ought to be different, but this seems to be pretty far from the present community standard.) $\endgroup$ Aug 25, 2010 at 19:48
  • 5
    $\begingroup$ In some fields, they PAY referees of papers, and thus have more leverage to ask them to do things. $\endgroup$ Oct 10, 2010 at 13:55
31
$\begingroup$

1) Should you summarize the main results and or the argument?

In general, I would say: No. However, for some journals I have come to know the overwhelmed editors who in fact need this summary. So: Write it for the editors, if you write it at all. [But see my acquiescence to Jukka Suomela's compelling point in the comments.]

2) What do you do when the journal requests you to evaluate the quality of the result (or its appropriateness for the journal)? Should one always make a recommendation regarding publication?

This requires you to know the standards of the journal in question. If you do, then definitely evaluate according to those standards. For example, I review for Discrete & Computational Geometry and for Geometriae Dedicata, which both have high standards and long backlogs. I take that into account. Does this paper deserved to be published in the all-too-few pages that these prestigious journals can provide each issue, rising above many others?

3) What to do about papers with major grammatical or syntactical errors, but that are otherwise correct? Does it matter if the author is clearly not a native English speaker?

I view my job as referee to evaluate both the quality of the mathematics and the comprehensibility of the writing. I sometimes will edit just one paragraph for style and clarity (with maybe ten what I think are improvements), and suggest that this be done throughout. But it is not the job of a referee to be a style-editor, nor a 2nd-language improver.

4) On this note, at [what] point does one correct such mistakes? Ever? If there are fewer than a dozen? Should one be proof-reading the paper?

I may be abnormal in this respect, but my referee reports are not uncommonly $\ge5$ pages in LaTeX.

5) What do you do when you do not understand an argument? Does it matter if it "feels" correct. How long should one spend trying to understand an argument?

Yes, it matters! Say it "feels correct" but you have not verified it in detail. The question of "how long" depends very much on the perceived quality of the paper. I have spent upwards of 50 hrs on what I thought was a seminal paper, but have dispensed with equally long papers in an hour or two if I thought their results were pedestrian (even if correct). [I originally said 100 hrs, but upon reflection I think it only felt like 100 hrs.]

6) What to do about papers that have no errors but whose exposition is hard to follow?

You say exactly that, detailing where and why. How to proceed from here is an editorial decision, not a referee decision.


Finally, I will say something perhaps controversial, but not apropos RBega's question. I think anonymous refereeing is funadmentally defective in the abuses it invites, and therefore I only review nonanonymously. But I am sufficiently senior to render this without risk.

$\endgroup$
8
  • 41
    $\begingroup$ Regarding (1): I think you should always summarise the main result. Besides the obvious (it shows that you have actually bothered to have a look at the paper before writing your review), I think it serves a useful purpose even in the case of a conscientious reviewer: it will show the authors how you interpreted their paper. The authors can't always communicate clearly what are their main results and exactly why they are interesting; reading the reviewer's summary of your own paper may be enlightening (and explain why the reviewer thought that the results aren't interesting). $\endgroup$ Aug 24, 2010 at 23:39
  • 4
    $\begingroup$ Re: the final remark, referees should not take too much comfort with the idea of anonymity. Most of the time, one can narrow down a referee's identity from the report. I wouldn't go out of my way to identify myself, but I wouldn't try to hide either. $\endgroup$ Aug 24, 2010 at 23:50
  • 3
    $\begingroup$ One paper of mine that - fortunately - got positive referee's reports came back to me with the reports attached as PDF files called "smith.pdf", "jones.pdf" and "bloggs.pdf" where {smith, jones, bloggs} were the referee's surnames. So I always write a report with the sense that my anonymity is far from secure - so I try to be positive and constructive even when recommending rejection. $\endgroup$ Aug 25, 2010 at 3:48
  • 4
    $\begingroup$ Re (2); this is the editor's job, not the referee's (who should not be second guessing the journal's backlog or policies). By making the judgement yourself you can make the editor's job harder, if they disagree with your recommendation. Better to say what other journals you would expect this paper to be published in or not published in. This establishes exactly what standard you think the paper is, and helps the editor, who can then make their own decision on whether to publish. $\endgroup$ Aug 25, 2010 at 6:20
  • 9
    $\begingroup$ @Richard Thomas: I quite disagree with you. First, I know journal that expects you to propose acceptance or rejection (and give you hints on the standards they want to hold and their backlog). Second, I don't see how one could be able to judge the standards of several journals so that she or he can give an opinion whether the paper could be accepted in those, while not being able to make the same for the very journal that he or she is refereeing for. $\endgroup$ Aug 25, 2010 at 8:12
26
$\begingroup$

The referee must check if the result is correct! That is, the referee should check the proofs. This point seems to be so under-estimated, but in my opinion it is the most important part of the referee's job. The referee is the only one who will do this, except the author. There does need to be a second check of correctness after the author. There are many flawed papers out there. This has cost many other researchers time and health.

If the referee is unsure, he should ask the author for help in understanding the proofs (and the author should be helpful). It can be a change in the paper or a note for the referee.

As for whether or not the paper is "interesting": I decline to referee a paper if it is not interesting (in a broad sense) enough for me to put the effort in checking whether or not it is correct. Thus if all referees were correctness minded, then "interesting" papers would get their advantage.

$\endgroup$
6
  • 1
    $\begingroup$ When you ask an author for help understanding a paper, how can you do that without sacrificing anonymity? Or is anonymity even important? $\endgroup$
    – Jim Conant
    Oct 10, 2010 at 13:38
  • $\begingroup$ caused--->cost? $\endgroup$ Oct 10, 2010 at 13:38
  • 3
    $\begingroup$ @Jim: perhaps through the editor? $\endgroup$ Oct 10, 2010 at 14:12
  • $\begingroup$ You can ask for clarifications via the editor (in some journals at least). Anyway, when refereeing is less judgemental ("this paper is correct but the results are not important"), as I see it, anonimity is less important. $\endgroup$
    – Orr Shalit
    Oct 10, 2010 at 14:14
  • 2
    $\begingroup$ Perhaps "judgemental" is bad wording. When you say a proof is not correct there is much less room for debate, usually. But saying that a result is not important is something that cannot be proved in the usual way we prove things. $\endgroup$
    – Orr Shalit
    Oct 11, 2010 at 14:07
22
$\begingroup$

Be timely (if you can't do it in <6 months then don't agree to referee). Be accurate. Don't take the author's reputation into account when rejecting or accepting an article, and insist on good exposition. Remember also that your report is a report to the editors, not the author.

$\endgroup$
3
  • 32
    $\begingroup$ I think the referee's report is for the editor and the author. I would be very annoyed if the editor of a rejected paper solicited a referee's report but did not send me a copy. $\endgroup$ Aug 25, 2010 at 6:42
  • 7
    $\begingroup$ It happened once to me that my paper was rejected (after 5 months) without sending me the report. I can tell you that it was annoying. $\endgroup$ Aug 25, 2010 at 15:35
  • 1
    $\begingroup$ That's a different issue -- the question is who the referee should view as his or her audience, not whether an editor should send the referee reports to the author (which of course he or she should). $\endgroup$
    – JBL
    Aug 25, 2010 at 17:15
17
$\begingroup$

I feel that I more or less agree with everything that has been said so far (so in a sense I don't know why write this), but personally, I find it really annoying when the type-setting and grammar are really really poor in a paper that I've accepted refereeing (maybe I shouldn't have accepted then you might say).

If the authors haven't spent the effort making the paper readable and pleasing to the eye it all looks so sloppy and that doesn't inspire confidence in the paper. I know this is wrong of me, but that's the way I feel.

In practice though (this has happened more than once) I try my very best to give the paper a fair and unbiased review and I also indicate in my report most of the typos and grammatical errors I find, even if this is somewhat pedantic. I feel that my job as a referee is also to make the paper readable to others, in addition to error-spotting.

$\endgroup$
1
  • 2
    $\begingroup$ If there are many typos and/or language problems, just pointing out in the report that major fixes are needed in that department ought to be enough. Journals should have people to fix that (conference proceedings are another matter). On the other hand, if the mistakes are not overwhelmingly numerous, I try to track them down and include them in the report, reasoning that the more people work on that, the fewer typos end up on the printed page. $\endgroup$ Aug 25, 2010 at 13:21
14
$\begingroup$

I am not sure if this point has already be raised: If your report criticize something or votes for a revision, then give hint what the authors can do. Comments of a referee which are particularly useless are e.g.: "The presentation is messy" or "The references are not appropriate".

$\endgroup$
11
$\begingroup$

Re question 5 ("How long should one spend trying to understand an argument?"): an interesting case was Hales's proof of the Kepler Conjecture, in which the 12-member team of Annals referees spent five years before resigning themselves to being unable to completely certify the validity of the proof.

$\endgroup$
1
  • 4
    $\begingroup$ And now Hales is embarked on "A Revision of the Proof of the Kepler Conjecture" [Discrete & Computational Geometry, Volume 44, Number 1, 1-34, July 2010]. $\endgroup$ Aug 26, 2010 at 19:30
5
$\begingroup$

I've had long discussions about whether the referee report should be author-dependent. If mathematician X and mathematician Y submit the same paper, should the report be the same?

$\endgroup$
10
  • 15
    $\begingroup$ If mathematician X fails to properly cite her own previous publications, that would probably be less of a sin than if mathematician Y failed to properly cite publications of mathematician X. $\endgroup$
    – Colin Reid
    Sep 8, 2010 at 16:28
  • 6
    $\begingroup$ @Colin, you comment seems to miss the point: if X fails to cite X, this is no different from Y failing to cite Y. $\endgroup$ Sep 9, 2010 at 1:31
  • 25
    $\begingroup$ I once handled, as an editor, a paper written by a postdoc, in which he had rediscovered some known results. The referee provided a beautiful report, not only explaining what was already known but suggesting related problems that were still open. Had the same paper been submitted by an established mathematician, the report might well have simply been "reject because the results are already in such and such papers." $\endgroup$ Sep 30, 2010 at 0:16
  • 7
    $\begingroup$ I fear I am misunderstanding how there could be a discussion about this. But after thinking about it a bit, I realize there are cases when a report (in my opinion) should depend on the author. The main example in my mind is when the author is a new mathematician. You can give advice (and they may be more likely to take it). You should be extra careful about being prompt if they are likely to be applying for jobs soon. ... $\endgroup$
    – Ravi Vakil
    Nov 3, 2010 at 21:07
  • 21
    $\begingroup$ ... You should always be kind, but you should be extra careful about being kind to someone starting out in their career. A blunt or brutal report can have a worse impact than you might think. $\endgroup$
    – Ravi Vakil
    Nov 3, 2010 at 21:07

Not the answer you're looking for? Browse other questions tagged or ask your own question.