Question

Let $X_i$ (where $i=1, \dots , n$) be independent and identically distributed 3d rotations. What is the distribution of $X_1X_2\dotsb X_n$ in the limit of large $n$?

I'm especially interested in the special case where $X_i$ have mean 0 and are highly concentrated i.e. they are very small rotations. Just to make this clear, an example of this in the 2d case would be the rotations $(x,y)\rightarrow(x\cos \epsilon -y\sin \epsilon, x\sin \epsilon +y\cos \epsilon)$ where $\epsilon$ is a random variable with mean 0 and small variance.

Answer 1

About your 2D example: for $\epsilon$ of fixed variance, as $n\to\infty$, the composition will converge in distribution to the uniform (= Haar) measure on the orthogonal group. In less fancy words, the angle of the rotation will converge to the uniform measure on the circle. Note that this will hold whether $\epsilon$ has mean $0$ or not, just some non-degeneracy (typically, the additive group generated by the support $\epsilon$ should not be a discrete sub-group of the circle ...) and positive variance are enough.

In the more general case, typically (meaning under reasonable assumptions about the distribution) if you compose rotations around the origin with fixed variance, you converge to a uniform distribution on the orthogonal group (the Haar measure).

There will be two more interesting setups: the affine case (i.e. the rotation center is not always the same) and the "CLT" case when you choose $\epsilon_n$ together with $n$, something like $1/\sqrt n$. The first one I am not sure about; for the second one, you will end up with Brownian motion on the orthogonal group and the limiting distribution should be the exponential of a random, Gaussian, anti-symmetric matrix (i.e. a Gaussian on the corresponding Lie algebra).