I'm not certain of the complete answer, but I think there's a rough classification as follows. If $H < SO_0(n,1)$, then either
- $H$ is compact and is conjugate into $O(n)$
- $H$ is solvable and preserves a 1-dimensional subspace on the light cone (i.e. an isotropic subspace). In this case, $H$ acts by conformal affine transformations of $\mathbb{R}^n$.
- $H$ preserves a 2-dimensional subspace of signature $(1,1)$ (this could actually be subsumed in the previous case). Here, the action restricted to this subspace is $SO_0(1,1)\cong \mathbb{R}$. Then $H$ acts on the orthogonal space by a subgroup of $O(n-1)$. The action is a direct product of a closed subgroup of $O(n-1)$ and $\mathbb{R}$. However, this product is not canonical, since any generator of the $\mathbb{R}$ factor may be modified by a 1-parameter subgroup of the $O(n-1)$ factor.
- $H$ fixes a higher dimensional subspace of signature $(k,1)$, $k>1$. Then $H$ splits as a product of $SO_0(k,1)$ and a compact subgroup of $O(n-k)$.
Classifying the compact subgroups of $O(n)$ is complicated, so I think this might the best one can say in general. I don't know a reference, and I hope I haven't overlooked any possibilities. The way I think about this is the action on hyperbolic $n$-space and its compactification by the sphere at infinity. Either the action fixes a point in $\mathbb{H}^n$, or it fixes a point at infinity, or it preserves a totally geodesic subspace of some dimension.