How to resolve a disagreement about a mathematical proof?

Question

I am having a problem which should not exist. I am reading what I believe to be an important paper by a person - let me call him/her $A$ - whom I believe to be a serious and talented mathematician. A lemma in this paper is proven by means of an argument which, if correct, is a highly elegant piece of mental acrobatics in the spirit of Grothendieck, where a complicated situation is reduced to a simple one by embedding the objects of study in much larger (but ultimately better) object. Unfortunately, the beauty of this argument is - for me - marred by a doubt about its correctness. In my eyes, the argument rests upon a confusion of two objects which are not equal and should not be, but have the same name by force of an abuse of notation going awry. A dozen of emails exchanged with $A$ did not clear up the situation, and I start feeling that this is unlikely to improve; what is likely is that after a few more mails the correspondence will degenerate into a flamewar (as any prolonged arguments with my participation seem to do, for some reasons unknown). The fact that $A$ is not a native English speaker adds to the difficulty.

At this point, I can think of several ways to proceed:

• Let go. There is a number of reasons for me not to choose this option; first of all, I really want to know whether the proof of the lemma is correct or not (even though there seems to be a different proof in literature, although not of that beauty), but this has also become, for me, a matter of idealism and an exercise in tenacity (in its cheapest manifestation - it's not like writing emails is hard work...).

• Construct a counterexample. This is complicated by the fact that I am attacking the proof, not the theorem (which seems to be correct). Yet I think I have done so, and $A$ failed to really address the counterexample. But given the frequent misunderstandings between us (not least because of the language barrier) I am not sure whether $A$ has realized that I am talking counterexamples at all - and whether there is a way to tell this without switching to what will be probably understood as an aggressive tone.

• Request $A$ to break down the argument into simple steps, eschewing abuse of notations. This means, in the particular case I am talking about, requesting $A$ to write two pages in his/her free time and respond to some irritating criticism of these pages with the prospect of seeing them destroyed by a counterexample. I am not sure this counts as courteous. Besides, the paper is about 10 years old - most authors do not even bother answering questions on their work of such age.

• Go public (by asking on MO or similarly). This is something I really want to avoid as long as there is no other way. Neither criticizing $A$ as a person/scientist, nor devaluing the paper (which consists of far more than the lemma in question...) is among my goals; besides I cannot rule out as improbable that the error is on my side (and my experience shows that even in cases when I could rule this out, it still often was on my side).

• Have a break and return to the question in a month or so. I am expecting to hear this (seems to be a popular answer to lots of questions...) yet I am not sure how this can be of any use.

These ideas are all I could come up with and none of them sounds like a good plan. What am I missing? Is my problem a common one, and if yes, does it have a time-tested solution? Can it be answered on this general scale? Is it a real problem or an artefact of my perception?

PS. This is being posted anonymously in order to preserve genericity (of the author and, more importantly, of $A$).

Answer 1

There are three separate issues here.

1) How to clarify whether the proof is correct? You should start with making a serious good will effort to understand what is written (which amounts to redoing all the bad notation, splitting things into small steps, etc. to the best of your abilities). If this fails, you should state as clearly as you can what exactly the problem with the argument is and hope that some expert will figure out who is right. Of course, first you should send the full account of your effort to the author reproducing all the parts of the proof you understand and showing clearly where you are stuck and why. Just to say "your notation is bad here so..." won't accomplish anything: at best, he'll make a local correction that will move the real issue somewhere else and you can play this shifting game forever.

If he still fails to address the issue after that, request the help of some third party sending the same account of your effort together with the paper. Again, it is important that you demonstrate your good will and decent understanding of what is written in the paper before you raise your objection. Without this, you just won't be taken seriously. Make sure that you understand everything that precedes the unclear/incorrect step and that you make it clear to everyone whom you want to ask that you understand it. Nobody pays attention to people coming out of nowhere with zero credentials and doubtful qualifications. If your first words are "I don't understand ... and I think it is wrong", the most likely answer will be "Go learn ... ". However if your first words are "This argument starts with using ... to establish ...", your general credibility goes up immediately (provided that what you are saying makes sense). The more times the person agrees with you on the issue before you raise the question, the more likely he is to take you and your objection seriously.

It is your moral duty to make a real effort trying to understand the proof before making any public comment on its correctness but it is also your moral duty to report a problem with a proof when you are convinced that you see one. Note that it is completely normal in mathematics to make bad mistakes occasionally and it is completely normal to fail to understand correct arguments now and then. The priority/reputation chase has distorted the general attitudes beyond recognition, of course, but the heart of the matter is still the search for the knowledge, not building/preserving/destroying reputations and relationships.

Even if you are wrong on the account that the proof has a gap, you may be right on the account that it is unclear (assuming that you have a decent education in the subject, the fact that you fail to understand the argument is a clear indication that the paper is written not in an ideal way). So, the clarification may help innumerable poor souls (like graduate students) who may have the same difficulty but just do not dare to ask questions. You risk to look like a fool, of course, but the only way to avoid looking like a fool occasionally is to always be one.

2) How to avoid the confrontation? At some point there may be no way and all that you'll be able to write to the author something like "It seems very difficult for us to understand each other. Since the issue is principal, the best we can do is to seek an opinion of a third party. I'd appreciate your suggestions of whom we should ask. I'm thinking of (put the list of experts you know)". This may not save your good relationships but, at least, will clear you from any "doing things behind the curtains" charges. After that, send your doubts to both the people on your list and the people on his list, if he provides one. If he doesn't, it is his problem. There are three possibilities: 1) you'll be backed by some expert, which will make the author harder to ignore you; 2) someone will explain to you why the proof is correct, and 3) everyone will ignore your request. In that last case you may have to seek the opinion of general public but not before you double check your argumentation.

3) Is MO an appropriate place for this discussion? It is not what it was intended for but if you finally decide to seek the general public opinion and post your objection on arXiv or somewhere else (in the way I outlined above; let me emphasize once more that unless you are Terry Tao or Tim Gowers you should demonstrate both good understanding of the matter and your good will before anyone will bother to take a look at your objection in honest) I see nothing wrong with making a short post containing the corresponding link in this thread.

In brief, if you really want to figure things out, I would advise that:

a) you make a good effort putting all your thoughts together in written. Create a clear "case" starting from the beginning where you agree with A on everything and talk in the same terms and stopping where you have an objection.

b) present this full writeup to A and wait for his comment.

c) if it doesn't result in anything meaningful, present this writeup to a few experts or, at least, people whom you feel to be more knowledgeable in the subject than yourself.

d) if you are totally ignored, ask yourself why that may be the case and, if you see nothing wrong with your written argumentation (if you see nothing wrong with what you keep in your mind, there is a good chance that you are just blind), present it to the general public.

I don't think asking a graduate student of A is a good idea. First, many graduate students are totally incompetent in anything beyond their thesis project and in such cases, you can just as well use a parrot for communication. Second, if the student is actually good and you are right, you'll put the student in the position where he will have to tell his adviser that the paper is wrong. This doesn't go well with many people.

Answer 2

I think you neglected the simplest options of all: talk to other people about it (privately, not on MO). Even if that other person is not an expert in the field, it is likely that you will be able to clarify the arguments for yourself, while trying to explain what is wrong with the proof.

Answer 3

A variant of your third point "Request A..." could also be an option.

That is you try to rewrite the proof, avoiding the abuse of notation but otherwise staying close to the original, and then point to a specific point where you have difficulty/see problems and ask A for clarification on something very specific.

This would be less work for A and also document to him/her that you have a very serious interest to understand the situation. Or, in the process you might notice that there actually is no problem.

Whether to wait is a good idea or not depends on the situation; at least I would tell A that you only interrupt the communication rather then stop it. Otherwise, it could be starnge if s/he misunderstands and thinks the problem is resolved and then you restart after some time.

Also the suggestion made by others to ask other people (privately) to me seems very good; indeed if it is feasible most likely best [but in view of the fact you ask, I could imagine this is a nonoption].

Answer 4

I would add something else. Talk to someone who has used this result before. Hopefully there will be someone and he/she will have read the proof. I think it is easy to find who has referenced a paper through mathscinet. Obviously this person should be accessible to you, but it worths a try (it could be someone you have worked with and you feel more comfortable with).

And keep being tactful :)

Answer 5

I would suggest a combination of the second and the third: construct your counterexample (to the lemma, not the theorem) in simple steps, eschewing abuse of notations, and showing where their proof would fail under both (each?) rigorous interpretation(s) of their notation if attempted to be applied to the counterexample.