It is a known fact that the adjoint representation of $SU(N)$ is a proper subgroup of $SO(N^2-1)$.
I would like to know how a generic $(N^2-1)\times (N^2-1)$ special ($det =1$), orthogonal matrix $O$ decomposes into an element of $SU(N)$ in the adjoint representation, $[Ad_U]$, times something else, $$ O=M\cdot Ad_U $$
Clearly $M$ will be an element from a group, but I cannot figure out which. My background in group theory is rather limited.
Actually, it does not look like that. Take the case $N=3$. The representation of $\mathrm{SU}(3)$ on ${\frak{so}}(8)$ breaks up into the $8$-dimensional subspace ${\frak{su}}(3)$ and an irreducible $20$-dimensional space that is isomorphic to the complex homogeneous cubic polynomials in $3$ variables as an $\mathrm{SU}(3)$ representation. That subspace is not a Lie subalgebra of ${\frak{so}}(8)$, so there is no $20$-dimensional Lie subgroup of $\mathrm{SO}(8)$ corresponding to it.
Maybe what you really want is something like an explicit $20$-dimensional manifold $M$ and an explicit map $\mathrm{SO}(8)\to M$ whose fibers are the left cosets of the subgroup $\mathrm{Ad}\bigl(\mathrm{SU}(3)\bigr)$? That would be more reasonable.
Here is another point to think about: There cannot be any closed subset $S\subset\mathrm{SO}(8)$ such that the map $S\times \mathrm{Ad}\bigl(\mathrm{SU}(3)\bigr)\to \mathrm{SO}(8)$ induced by multiplication is $1$-to-$1$ and onto. The reason is that such a map would be continuous (using the product topology on the domain), and a continuous bijection from a compact space to a Hausdorff space is a homeomorphism. However this would imply that $$ \pi_1\bigl(\mathrm{SO}(8)\bigr) \simeq \pi_1\bigl(S\bigr)\times \pi_1\bigl(\mathrm{Ad}\bigl(\mathrm{SU}(3)\bigr)\bigr), $$ and this is impossible because $\pi_1\bigl(\mathrm{SO}(8)\bigr)\simeq \mathbb{Z}_2$ while $\pi_1\bigl(\mathrm{Ad}\bigl(\mathrm{SU}(3)\bigr)\bigr)\simeq\mathbb{Z}_3$. Thus, there cannot be any reasonable 'factorization' of the kind you are seeking, even if you don't require that the first factor lie in a Lie subgroup. (This argument generalizes immediately to cover all the cases when $N\ge 3$. Obviously, for $N=2$, you can take $S = \{I_2\}$.)
(Maybe, you were imagining something like the so-called $QR$-factorization of each element of $\mathrm{SL}(n,\mathbb{R})$ into the product of a positive definite symmetric matrix and an orthogonal matrix. Unfortunately, nothing like that works in this case.)
