How bad is it to publish a paper with an overcomplicated proof?

Question

I don't want to go into details for anonymity purposes, but I have co-authored and submitted a paper with a long proof (dozens of pages), and I think that with some moderate effort, we could find a proof that is about half as long, by decreasing the power of some intermediate lemmas (the reason for this is that we initially didn't know that it could be done more simply). How bad is it to publish a paper like that? Is this considered bad style? Or actually unethical? Is it considered embarrassing (e.g., will we be considered amateurs for not having found the simpler proof first)? Or is there nothing wrong with it?

Answer 1

This kind of situation is very common. In some sense, it's the nature of research. Consider the following quote from Andrew Wiles:

"Perhaps I could best describe my experience of doing mathematics in terms of entering a dark mansion. You go into the first room and it's dark, completely dark. You stumble around, bumping into the furniture. Gradually, you learn where each piece of furniture is. And finally, after six months or so, you find the light switch and turn it on. Suddenly, it's all illuminated and you can see exactly where you were. Then you enter the next dark room..."

The path you took from the door to the light switch was probably not the most optimal path. Once things are illuminated, you can go back and find a more optimal route. That said, you should not feel like it has to be the shortest, cleanest proof possible. Many times, you will discover years later that something could have been done in a shorter or cleaner way, and that's natural when you sit with ideas for a long time. You should not delay writing things up or getting things published, just because later you might find a cleaner way to do it. If/when you realize things in a cleaner way, then you can write a follow-up paper subsuming your original paper. That's the standard way to have a productive research career. Now, on to the OP's specific questions:

we could find a proof that is about half as long, by decreasing the power of some intermediate lemmas

As has already been pointed out in the comments, the stronger lemmas might be interesting in their own right. You yourself might need the stronger lemmas in your next paper. It's probably best to leave them, unless you get a referee report that insists on a much shorter proof.

How bad is it to publish a paper like that?

It's not bad. If the paper is good enough to be published, then publish it! Sometimes you realize things could be done in a simpler way after you've already heard back from the referee. It's unwise to rewrite the paper at such a late stage. Just plan to write a follow-up, or a book years down the line, with your newer and simpler approach.

Is this considered bad style? ... Is it considered embarrassing (e.g. will we be considered amateurs for not having found the simpler proof first)?

It's possible, but very unlikely. Once in a while, a senior person might read a paper and think "this could have been done in an easier way." But, senior people also know what it's like to be a junior person (and one who needs publications and doesn't have time to polish every proof to perfection) and would not hold it against the author. Presumably, as you go on, your proofs will be more directly to the point, and any suboptimality of early work will be forgotten.

Or actually unethical?

Certainly not. Remember that others might need those stronger lemmas. And having your paper, even if it's not perfect, is a whole lot better than having no paper at all proving this result.

Early on in your career, it's important to get preprints out the door (i.e., submitted), even if you think you could tinker with them and make them slightly better. You need every publication you can get. There will be plenty of time later, once you have tenure, to write things down in a nice, clean way that sheds as much light as possible. Early on, only rewrite a paper if it's clearly flawed, or if you must rewrite it to get it accepted for publication. It's better to err on the side of submitting something too early (and letting the referee tell you how it could have been better) than to sit on it and submit it too late to help you get a job.