Don't to do this problem. Very boring and you will learn nothing from it.

If the first advise does not work:

• First guess what is the answer,
• Second prove that it is a strict local maximum in the space of relative orientations of these two bodies; it is essentially SO(3). (This part should be nearly a calculus problem.)
• Finally, show that it is global maximum by estimating the value at a $\varepsilon$-net of SO(3) for small enough $\varepsilon$. (While doing this you may also learn that your guess was wrong and it will suggest a better guess.)

Minkowski subtraction might help to do the last two steps.

The computers now are better than in 80's so all this might work.

Don't do this problem. Very boring and you will learn nothing from it.

If the first advise does not work:

• First guess what is the answer,
• Second prove that it is a strict local maximum in the space of relative orientations of these two bodies; it is essentially SO(3). (This part should be nearly a calculus problem.)
• Finally, show that it is global maximum by estimating the value at a $\varepsilon$-net of SO(3) for small enough $\varepsilon$. (While doing this you may also learn that your guess was wrong and it will suggest a better guess.)

Minkowski subtraction might help to do the last two steps.

The computers now are better than in 80's so all this might work.