Consider two black holes with masses $m_1,m_2$ and zero angular momenta merging to form a single one with the mass $m$ and the rotation parameter $a=J/m$. Hawking, in "Black Holes in General Relativity" Commun. math. Phys. 25 (1972), 152—166 proposed an inequality $$m^2+m\sqrt{m^2-a^2}>2(m_1^2+m_2^2)$$ for this process (in fact, for a more general one, see p. 14 of the paper). I learned about this bound ages ago from the Lightman-Press-Price-Teukolsky relativity problem book and had no doubt about it. But now I think that the proof given in this paper is total rubbish despite being published in a supposedly mathematical journal.
The inequality is derived from what is now called an area theorem which sates that the area of the event horizon never decreases. There is nothing wrong with the theorem itself except the way it is formulated makes it completely useless for obtaining an inequality of this sort. (And probably for any other meaningful conclusion.) The fishy point here is the assumption that the area of a black hole event horizon is given by the formula (in geometric units $c=G=1$) $$A=8\pi m(m+\sqrt{m^2-a^2}).$$ No doubt, this assumption is true for a Kerr black hole but there is a big problem. The event horizon as it is defined in the formulation of the area theorem depends on the (arbitrarily distant) future evolution of a black hole, so even it the thing looks exactly as a standard Kerr black hole now its event horizon may still well be very different from what one of a Kerr hole is supposed to be, with very different area. There is no formula for the actual area of this event horizon in terms of the mass and the angular momentum.
To see where the problem really lies it is convenient to consider a scattering of two black holes instead of their merger. This process has an inverse which is also perfectly physical even if not likely to ever happen in reality. (Because general relativity dynamics is, of course, time-symmetric.) Then exactly the same argument as in the paper when applied to both processes gives two inequalities which contradict each other.
Admittedly, from reading more recent physical literature I have the impression that the problem is more or less known. However, it is never mentioned explicitly. Apparently, physicists believe that the inequality is true anyway and do not care much about gaps in its proof. A mathematician like myself would rather like to see an actual proof though. Is such a proof already known or, at the very least, was the problem ever considered seriously? This is my question.
