65
$\begingroup$

I have always checked very carefully the papers I was refereeing when I wanted to suggest "accept". Actually I spend almost as much time checking the maths of a paper I referee than checking the maths of a paper of mine (and this is very long !). But I have some doubts. Is it really my job as a referee ?

This question is related to Refereeing a Paper but only a few comments were made on that point in the above cited thread (mainly Refereeing a Paper).

$\endgroup$
3
  • 18
    $\begingroup$ I think that this is actually a bit vague. There's "What ought the job of refereeing be (in an ideal world)?", there's "What is the actual job of a referee?" (which is unanswerable), and "What is the job of a referee for a particular journal?" which has the answer: "If you're not sure, ask the editor who referred it to you.". $\endgroup$ Oct 1, 2010 at 11:05
  • 2
    $\begingroup$ This is only somewhat related, so it is a comment. There is clearly some variation of opinion on the extent to which a referee's job is to check proofs, or to which a referee/editor bears responsibility for errors in published work, and so on. But whatever your opinion: if you decide to reject a paper on the grounds that its results are largely contained in previously published work, you should make reasonably sure that the work you refer to is free from error. At the very least, check Mathscinet to see if the journal has posted a retraction. Sadly, I speak from multiple experiences. $\endgroup$
    – anon
    Dec 12, 2010 at 23:54
  • 2
    $\begingroup$ @anon: somewhat late, I would like to point out that it is a reasonable division of labor to consider that it is the author's job to point out any problem with previous arguments. This is part of the "why is my work interesting" job one has to do when writing an article; moreover, it is important that anyone who could think the result was already proven has access to this information. That being said, at least a basic check from the referee is preferable. $\endgroup$ Dec 16, 2014 at 13:42

6 Answers 6

75
$\begingroup$

I only hope that computer-aided proof checking saves mathematics before it collapses under the weight of decades of irresponsible publishing. Of all disciplines, peer review in mathematics should serve to guarantee nearly absolute confidence in the validity of published results. Many subjects have grown so complex that one can't reasonably expect new people coming to the field to take responsibility for the correctness of all the literature that they might need to quote.

I remember attending a seminar at a famous institute where a young speaker justified a step by citing a paper by a well-known and well-published worker in the field. A very, very famous mathematician in the audience, the recognized leader of the discipline, stopped him and said "I wouldn't believe anything in [so and so]'s papers." A hush went around the room. Later I asked a colleague of the impugned mathematician (a member of the same department and an expert in the same field) about the incident. "Yeah, everyone knows his papers are garbage" he said. I asked why they get published. "No one wants a fight. We publish them and then ignore them."

I don't want this sort of practice to define mathematics in the public mind. I think we should compensate referees for their hard work, and honor solid refereeing nearly as much as we do excellent research. Either that, or fund computer-aided proof checking to the hilt, change the methodology of the subject and get human beings out of the business of vouchsafing the literature.

$\endgroup$
6
  • 50
    $\begingroup$ I am of course very curious now. $\endgroup$ Dec 11, 2010 at 23:10
  • 10
    $\begingroup$ "I think we should compensate referees for their hard work" IF they do hard work. Unfortunately this doesn't seem to be the case. In my experience, over 50% of mathematical papers in renowned journals have notable flaws (not misprinted letters, but actual holes in proofs, although usually fixable by any expert in the field; also, definitions that don't match the actual later use of the notion defined). Unless I really care about some result, I don't even try to read it up in a research paper - I wait until a book or, at least, a review article, appears. $\endgroup$ Dec 11, 2010 at 23:46
  • 25
    $\begingroup$ It's worth saying that I've never heard this kind of story in any area of mathematics I've been connected with, and I believe it to be highly unrepresentative of the current state of publishing in pure math. $\endgroup$
    – JSE
    Jan 27, 2011 at 16:21
  • 7
    $\begingroup$ You should disclose the name of the "garbage printer" so that we can also ignore him. $\endgroup$
    – Najdorf
    Jan 27, 2011 at 17:11
  • 12
    $\begingroup$ I can think offhand of at least 3 different mathematicians who I think do great work, but whose papers and books are extremely unreliable. The ideas are great, but the details are often all wrong. It's a tricky dance figuring out how to cite them. $\endgroup$ Dec 16, 2014 at 20:14
31
$\begingroup$

I regard it as part of my job as a referee. But the amount of time I spend checking proofs really depends on whether the point of the paper is to prove something I already believed but didn't know how to prove (in which case I spend a lot of time) or to tell me something new, in which case I might spend very little time on the proofs and rather focus on deciding how interesting the new facts are.

I agree with the oft-expressed sentiment that it is primarily the responsibility of the author to check for correctness. Editors may well find your assessment of the value of the theorems contained in the paper more useful than your assurances that the proofs are correct.

Someone who is interested enough in the results of a paper to use them is going to be the most likely source of corrections for the proofs. I've found more errors in papers whose results I needed to apply than in papers I've officially refereed.


Edit: I think it's worth responding to David Feldman's answer. I agree with his aim: ensure that mathematics literature is not full of errors. But I think that the refereeing process is not the most efficient means for weeding out inaccuracies. Better for that to happen organically as the consumers of new ideas put them to the test. The arxiv helps quite a bit, by ensuring that ideas are disseminated quickly to those that are likely to appreciate them. Why centralize this process? Furthermore, even with the most conscientious refereeing, mistakes will slip through if the only two people who have read the paper carefully are the author and a reviewer (and if those really are the only two people who have read the paper carefully, then it's probably not a big deal that an error made it through, anyway).

Roy Smith's comment below is a good one: if you don't have time to check the proofs carefully, tell the editor that. S(he) can then make an informed decision about publication. Often I receive a request to referee a paper that I am sure I will find useful at some point in the future, but I don't have time to check carefully all the proofs (real life and other work can get in the way). I could tell the editor to find another referee, or I could do the best I can in the time I have. Sometimes I really think it's better for the mathematical community to choose the latter, since finding someone to referee a paper can sometimes be a real timesink.

Perhaps the disagreement here is a cultural one: some people think of mathematics as an experimental science, some think of it as primarily about finding the right answers, and some think of it as primarily about finding proofs. In the last case it's natural to put a premium on proof checking.

$\endgroup$
6
  • 15
    $\begingroup$ I don't disagree. However, the danger I see with that approach is the emergence of consensus mathematics. The stuff that looks right does not get scrutinized much, which, even if the results are true, does not nurture progress. All you need is to read the book en.wikipedia.org/wiki/Proofs_and_Refutations to realize that an "obvious" result like the Euler characteristic of a polyhedron benefits from intense scrutiny. $\endgroup$ Oct 1, 2010 at 18:29
  • 9
    $\begingroup$ Thierry, it seems to me that what you are saying is the opposite of what is suggested above: the OP says that he or she scrutinizes the proofs of things he or she believes more. $\endgroup$
    – JBL
    Oct 1, 2010 at 19:44
  • 1
    $\begingroup$ Much of physical science builds upon the results of others. The only ones who will rigorously test and check and validate the results in a particular subtopic, even in mathematics, are those whose interests contain that particular subtopic. Those who would like to apply and extend the results are thus most likely to find the flaws in it. I would guess that this is why referees for a paper should be those who work in the particular subtopic or field of the article being reviewed, and if they're interested in extending those results, they'd scrutinize the article carefully. $\endgroup$ Oct 2, 2010 at 10:26
  • 3
    $\begingroup$ @JBL: I was adding a point about the stuff that I already believed and already believed I knew how to prove. I've run across those a couple of times when I see a lemma and think: "oh, sure, you do X and Y and it'll work out." But neglecting "routine arguments" can make us miss their unexpected depth, a remark that does not have to do much with the good of the refereeing process as to the health of mathematics in general. $\endgroup$ Dec 12, 2010 at 18:13
  • 3
    $\begingroup$ If you do have time to check everything, that is a valuable service to everyone. If not, just say so. I have found gaps in arguments by very strong people, usually by citing other famous people erroneously, after months of checking. In a few cases I have told the editor the results were important, the arguments looked careful and thorough but I had not checked them all. It is the editor's job to decide whether to publish or re - referee based on your remarks. The less you check, the sooner you want to finish of course, within reason. $\endgroup$
    – roy smith
    Dec 12, 2010 at 19:11
21
$\begingroup$

I agree with David: I think that my job as a referee consists mainly in checking that the proofs are correct.

However, checking that the proof is correct can in practice not be done by line-by-line checking. Referees are not computers, nor are the writers of the articles. Rather I have a kind of critical, "falsificationist" approach. I try to see what are the principal steps of the proof. If the article is well-written, this work has been done by the authors. Each of those stepsis an assertion, and I try to disporve it. Is it in contradiction with something I know? Can I find a counter-example? If I can't I begin to believe a little in this one step, so I try to prove it myself. If it's too hard, I read the proof of the argument (if there is one!) and apply th same method th understand and check it.

$\endgroup$
2
  • 3
    $\begingroup$ And if the proofs are all flawless, do you always recommend acceptance? How do you decide whether or not to review an article in the first place? Do you really not spend much time deciding whether the results are significantly different from what is currently available? This last task is where an expert referee can really earn her (non-existent) keep. A good graduate student can often provide just as critical a reading of the proofs. $\endgroup$ Dec 12, 2010 at 17:33
  • 3
    $\begingroup$ @Sheikrainrollbank: you have asked a bunch of nontrivial questions here. It might be worth asking them again in a more prominent place: e.g. editing them into the question, or asking as a new question. Just as a comment on Joël's answer: perhaps he does not mean to imply that judging the context / novelty / significance of the results is less important than reading for correctness, only that the latter task is much more time consuming (hence is "most of the job" in some sense). $\endgroup$ Jan 27, 2011 at 19:46
15
$\begingroup$

I more or less agree with Sheikraisinrollbank, but for the sake of argument...

I've always found it slightly hard to fully separate "Is this paper interesting" from "Is this paper correct". It seems like mathematics (especially towards the pure end) is full of interesting, plausible "facts" we don't have proofs of. Isn't part of the point of maths to find rigourous proofs for things? So if I want to claim that a paper is interesting, it seems a priori necessary to have some faith in the paper being correct!

Similarly, the "interest" of a paper, to me, is often bound up in the methods of proofs being used, not just the statements of the theorems (is this proof something which I would have tried if I'd thought hard, or is it completely from left-field?) So I'd want to read the proofs carefully, even if I wasn't "checking" them.

So, if had to advise: Yes, you should read the proofs, closely. But checking every line? Perhaps not, unless something looks very off.

$\endgroup$
11
$\begingroup$

I believe that the referee should check the proof is correct, or at least that the logical deduction of its claimed results from the assumed results it quotes is correct. It probably is unreasonable to expect referees to check the correctness of cited results in a paper- this could easily become a never-ending task. While there is probably a general consensus that responsibility for correctness rests primarily with the author(s), the community needs safeguards to ensure that genuine mistakes by authors are caught before they become assimilated into the body of accepted mathematical knowledge.

However, I also believe that authors who quote previous results in their own proof have a responsibility to ensure as far as possible that they fully understand the results they are quoting, and their correctness. If a quoted result turns out to be wrong, the person who quoted it can't evade their own responsibility by blaming the referee of the paper quoted.

Of course there is a conflict between idealised behaviour, and what is attainable in the "real" world, but in the long run, Mathematics, perhaps above all other human endeavours, strives for an idealized perfection, and abandoning that striving would be fatal for the subject.

$\endgroup$
6
$\begingroup$

I am waiting for a time when high ranked journals will require besides mathematical proofs written for human also alternative proofs that can be checked by computers (written in special language like Coq or Mizar). In this case referee will not have the obligation to check the validity of the proofs (this will be done automatically at the submission stage) but concentrate on evaluation of the importance of obtained results etc. This would save a lot of referee's valuable time and simultaneously will guarantee that the papers accepted for publication do not contain errors.

$\endgroup$
3
  • 4
    $\begingroup$ How about the time of the authors? $\endgroup$ May 3, 2013 at 14:52
  • 13
    $\begingroup$ You are going to wait a long time! $\endgroup$ Jun 13, 2013 at 22:31
  • $\begingroup$ We should indeed recognize the importance of proof checking, using any tool at our disposal, but if we want to progress in our understanding of mathematics we also need to continue to have human-read (not only human-readable) proofs. $\endgroup$ Dec 16, 2014 at 13:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.