Should I publish a paper if its results overlap significantly with an earlier paper?

Question

I have a preprint X that is sitting in the ArXiv for which I am not sure if it is still worth publishing. It turns out the paper I wrote has considerable overlap with another preprint Y after one of its authors informed me about it through email. Consider the following:

1. Paper Y was posted in the ArXiv just a month before I posted mine, and I was not aware of its existence previously. Prior to posting paper X, I looked for as many papers with related results to include in my discussion, but due to the differences in the terminologies used, Google did not show paper Y in the results. In order to find paper Y on Google or ArXiv, one would have to use a different set of keywords. It appears that we have been working on the same problem, they just finished first and had a month of lead.

2. I am aware that alternate proofs are sometimes of interest and are therefore worth publishing, and I am trying to determine if this is the case with my paper. Since I wrote paper X independently and without knowledge of the existence of paper Y, my proof was essentially different and indeed the two papers have very different motivations for the constructions used. However, while the methods used as different, they have the same "flavor" and the key results in both papers use different versions of the same pre-existing theorem.

3. If there is anything drastically different from the results of both papers, it would be the length. My proofs are shorter than theirs, use less lemmas and mathematical machinery. Paper Y derives a few more corrolaries which are not in paper X, but even if one compares the length of both papers using only the parallel or similar portions, my paper is still significantly shorter than theirs. I can also make the argument, understandably a subjective one, that my methods are simpler. However, results of Paper Y are, to a certain degree, more general than mine and therefore, technically my results follow from theirs.

My questions are as follows:

1. Should I still submit my paper to a journal?

2. If yes, how should I deal with the existence of paper Y in my own paper? I feel obliged to cite it, but I'm not really sure how to discuss the similarity and differences between the results. Should I tell the editor about the situation?

3. Should I make other assumptions on the reason behind the author of Paper Y for sending me an email about their paper, other than to inform me that our papers have the same results? If you were in the shoes of the author of paper Y, what would you prefer that I do?

Answer 1

I once wrote a paper with an undergraduate that I thought was very nice. After it was accepted for publication, we found a paper not only proving our results, but going a step further. We hadn't found it previously because, similar to your situation, they used different terminology. In our case, our proofs didn't add anything new, and our results were weaker, so we withdrew the paper before it was published. That same undergraduate went on to write more papers with me, and is currently a successful graduate student. I mention this to emphasize that not publishing is not the end of the world.

If you decide to publish, you should definitely cite the work you found, and mention to the reader that their proofs are different and came first. Something like the following should be fine: "After a preprint version of this paper was made public I was made aware of [Y], which appeared first and obtained Theorems ###. As the methods this paper are completely different, we believe that the proofs still merit publication."

It is not necessarily your job to explain those differences, unless you think it would help the reader. However, it is your job to convince (yourself, the readers, and) the referee that your paper is worthwhile. So you need to deeply understand the differences, and perhaps explicitly point out what your paper adds to the literature.

I imagine that the authors of Y mentioned their paper because they have priority, and would appreciate citations to their work. After you have cited their work, mentioning their priority, you owe no more obligations to them. (Although, if they contact you again, you can treat their communications with respect!) Don't try to mind-read their intentions. Focus on your own beliefs, whether you personally think your proofs would help readers understand the topic more deeply and give them interesting techniques. If you like, you can send Y an updated version of your paper, which clearly gives them priority, and kindly ask if they have any further comments.

Answer 2

In the few times in my career I have been in a similar situation, I have withdrawn my paper before publication, but gone on to modify it to be different enough from the other's (and my) prior work that it justifies publication. The lesson is: don't always view a paper as a complete, fixed, immutable work of scholarship but instead a working space, amenable to revision and improvement. Viewing your work this way may prod you to discover a new result (or a new approach to a known result) so your prior work wasn't "in vain."