Let us assume we have two 3D point sets, $P=\{p_i\}$ and $Q=\{q_i\}$, and that we need to recover the transformation that takes $P$ as close to $Q$ as possible. In particular, I am interested in roto-translations. In other words, I would like to compute the rotation matrix $A$ and the translation vector $v$ such that $q_i\approx A p_i + v$ (no scaling).
To be precise, I would like to obtain the best transformation in terms of least-squares (I am open to other criteria, though): $$\min_{R,v} \sum_{k=1}^n \|q_i-Rp_i-v\|^2 $$
I recently came up with the following algorithm to compute both $R$ and $v$, based on quaternions: Closed-form solution of absolute orientation using unit quaternions (by Berthold K.P. Horn, 1987), which seems to be quite popular in this matter. In fact, it works really well for part of the problem I am trying to solve. By the way, they also developed the orthonormal matrices version: Closed-form solution of absolute orientation using orthonormal matrices (by B.K.P. Horn, H.M. Hilden and S. Negahdaripour, 1988).
However, now I need to constrain the set of possible rotations or translations. For instance, by allowing rotations only on the XY plane, or translations in the Z direction...
The case where no rotation is allowed (transformations of the form $p_i-v$) is quite straightforward. For the pure rotation case (transformations of the form $Rp_i$, where $R$ corresponds to a rotation in any direction), I found a Matlab script where they use the same algorithm but without moving the centers of mass of the point sets towards the origin before computing the rotation matrix.
But for the rest of the cases, I have not found anything. Moreover, since Horn's algorithm mixes matrices with quaternions and eigenvalues, I have not been able to figure out how to modify it to include the restrictions. I would really appreciate any kind of help: references, algorithms...
