Suppose that I have $1000$ pair of points where each pair consists of a point in $\mathbb{R}^3$ and its image after a rotation in $\mathrm{SO}(3)$ with some noise. I have used RANSAC to find the matrix representation of the rotation (which is unknown to us) and since the number of inliers is not very large, the answer is accurate up to $2$ decimal points in each entry only. I have used the $L^2$ norm to determine inliers.
It turns out that even though the matrix representation is accurate up to $2$ decimal points, the axis-angle representation is highly unstable. I have tried to implement RANSAC using the axis-angle representation from scratch, and I have also tried to implement it using the matrix representation and then convert the answer to the axis-angle repesentation. Each time I run the algorithm, I get a direction that looks almost completely different from the last time. The angle is more or less the same, but the direction of the axis is very unstable.
I suppose that I need to use a different cost function for determining inliers. Is there a cost function that is known to give better numerically stable results for the axis-angle representation of rotations in $\mathbb{R}^3$?
Addendum:
It turns out that if I use RANSAC with the matrix representation of rotations and then I find the eigenvectors of the solution to find the direction (the direction has to be the eigenvector that is a real vector), it is stable. However, if I use RANSAC with the axis-angle representation, the result is still highly unstable.