After the uniform distribution (normalized Haar measure), the matrix Fisher distribution seems to be the most popular probability distribution on the Lie group SO(3). The density is proportional to exp tr(K MT X), but therewhere M is the mode and K (symmetric, often positive-definite) is the concentration. There are two ways to fill in the details:
- SO(3) is essentially the 3x2 matrices X such that XTX = I2, since the last column is determined by the other two. So M is 3x2 and K is 2x2. See e.g. Downs (1972).
- O(3) is the 3x3 matrices X such that XTX = I3. Here, M is 3x3 and K is 3x3. Restrict this distribution to the SO(3) component (and, and renormalize). See e.g. Chang and Rivest (2001), Kent et al. (2013).