Don't do this problem. Very boring and you will learn nothing from it. ---- If the first advise does not work: - First guess what is the answer, - Second prove that it is a strict local maximum in the space of relative orientations of these two bodies; it is essentially SO(3). (This part should be nearly a calculus problem.) - Finally, show that it is global maximum by estimating the value at a $\varepsilon$-net of SO(3) for small enough $\varepsilon$. (While doing this you may also learn that your guess was wrong and it will suggest a better guess.) Minkowski subtraction might help to do the last two steps. The computers now are better than in 80's so all this might work.