Getting nervous refereeing a paper I am refereeing my first paper and I'm quite excited! But inexperienced and I would like to ask an advice to the Maths Community of MO. Let me tell you that I have already read Refereeing a Paper, but it seems that my question is quite different. Roughly:

What is the point after which you get nervous while refereeing a paper?

Specifically:


*

*I've found many English mistakes. (Well, I'm not a native English and so I can understand. So I am not nervous yet)

*I've found some maths inaccuracies like "let A be any set"... and then I have discovered that the proof of the first result works only for finite sets. (OK, those are only inaccuracies - I am not nervous yet)

*There are many references like "we use the notation of [X]", "this result is proved in [X]", where [X] is a BOOK, without specifying a precise section, or the number of the result... should I get this book and read all to find out the correct references? - just thinking of it, makes me a bit nervous..

*(most importantly). There is a mathematical more serious mistake. Something that might be fixed, but not obviously (my definition of obvious is three evenings, in this case). I am not saying that the paper is completely wrong but that now... now I'm getting nervous!
Now, taking into account that the person who asked me to referee this paper told me: be selective, we accept only 20% of submitted papers, 

what should you do in these cases? Reject? Ask for a revision? Not getting nervous and try to see if the rest of the piece is good (I'm quite a good guy and I'm doing that at the moment)?

Of course I will talk with the editor, but I also would like to know more opinions that might be helpful in future.
Update: Thank you very much to everybody for the numerous and helpful comments.
Valerio
 A: This is really a comment but I don't have enough reputation points for that. It is a little more specific than Joël's.
It's a small world. How many papers are out there where the author defines some type of supremum of a set of elements in a group? Perhaps you have disguised the actual mathematical structure.
A: My suggestion would be to 
a. See if the claimed result is interesting. If not, reject.
b. If the claimed result is clearly interesting, be honest and say that the paper is clearly not publishable as is (and give your points 1-4, though really already 1 is sufficient), and should be revised before refereeing (this is often a box you can check if use one of the semi-automated review submission systems), but you would be willing to look at it once it is so revised.
c. Only get nervous over your own papers.
A: Others have already answered your question well, but I want to add something about #4, the mathematically more serious mistake.  Since by your own admission you are inexperienced, I can see why you spent three evenings trying to repair the proof, but I personally would never spend that much time in such a situation.  In my report, I would simply say that the proof is not acceptable in its current form because claim X is insufficiently justified.  (I would also keep going through the proof under the tentative assumption that claim X is correct, to see if there are any other issues later on in the proof.)  Say that the author must either provide more justification of X or find a way around it or retract the theorem.  In one case where I did this, the author did indeed retract the theorem (it was not the main result) because he was unable to repair the proof.
Proofs are supposed to be verifiable by competent readers with only modest effort.  If you're qualified enough to referee the paper then you should not be working so hard to understand the proof.
A: It is the author's job to be comprehensible, clear, and correct. If the paper falls short in that regard, you are doing the author and the editor a favor by asking for clarifications.
If you are feeling at all unsure, you could always say things like "I'm afraid I got a bit confused by Y", "I was not able to see how the proof of Lemma 1 handled infinite sets, would you please elaborate", "Would you please say where in [X] result Z can be found?", etc.
A: 3-4. Do not hesitate to ask the author (via the editor) what he had in mind.
A: I like the comment of Francis Crick in `What mad pursuit' that one should always start a referee report with praise, but of course not hold back on the detailed criticism. 
I have found that a referee can also help the author by asking for clarification in the Introduction of the aims of the paper, what does it accomplish in relation to previous work in the area, and what questions are still open. The author may also not have got the title right in relation to the expectations of readers: some of my early papers had lousy titles which not not convey easily what was in the paper. 
It is useful to know that the referee is not responsible for the correctness of the paper: that is the responsibility of the author. Of course, if the referee has doubts on the correctness,  these must be raised. 
I am unhappy about remarks such as "there are doubts about the interest of this area", since some papers are of interest precisely because they go against the so-called `mainstream', which, to mix a metaphor, flaps about like a sail in a gusty wind. 
The job of referee is very important, and I wish you luck and interest in this. 
A: Point (3): I would ask the author to be more specific, unless the results are basic stuff. People have asked me in the past to add a theorem number or at least a section number.
Point (2) and (4): that's exactly what goes in your referee report. Point (2) is a minor change, point (4) is a major change. List them all and send them back in a nice list.
Point (1): I would point out briefly that the English needs a review; I think proofreading the English should be a job for mr. Springer or mr. Elsevier after acceptance.
A: You can do the job of a referee, and you can add to it if you wish the job of a copyeditor/editor.  I think the criticisms you made all address the readability of the paper; if the author cares, he/she will incorporate your remarks.  The ones that matter to the publisher are the referee remarks, and you should ask your advisor/community how to present those.  As mentioned by others, the paper should be correct and interesting (and readable).  Be clear about what is needed to achieve those goals.  After that , add what reasonable efforts you believe the author should do to make the paper more readable.
Gerhard "Ask Me About System Design" Paseman, 2011.11.30
