# Numerical instability of the axis-angle representation of rotations in 3D

Suppose that I have $$1000$$ pair of points where each pair consists of a point in $$\mathbb{R}^3$$ and its image after a rotation in $$\mathrm{SO}(3)$$ with some noise. I have used RANSAC to find the matrix representation of the rotation (which is unknown to us) and since the number of inliers is not very large, the answer is accurate up to $$2$$ decimal points in each entry only. I have used the $$L^2$$ norm to determine inliers.

It turns out that even though the matrix representation is accurate up to $$2$$ decimal points, the axis-angle representation is highly unstable. I have tried to implement RANSAC using the axis-angle representation from scratch, and I have also tried to implement it using the matrix representation and then convert the answer to the axis-angle repesentation. Each time I run the algorithm, I get a direction that looks almost completely different from the last time. The angle is more or less the same, but the direction of the axis is very unstable.

I suppose that I need to use a different cost function for determining inliers. Is there a cost function that is known to give better numerically stable results for the axis-angle representation of rotations in $$\mathbb{R}^3$$?

You might like to try to enforce orthogonality of the matrix early on, and this seems possible: say, you have a pair of points $$u,v$$ and the unknown matrix $$A$$ so that $$Au=v$$. Now, if $$A$$ is orthogonal, i.e. $$A^\top A=AA^\top=I$$, then from $$A^\top Au=u=A^\top v$$ you get more conditions on $$A$$, for free. More precisely, you get 6 conditions from 1 pair of points, and from 1.5 pairs, so to speak, you should be able to get your 9 entries of an orthogonal $$A$$.