Now that you've added the maximality assumption, $\langle G,h\rangle$ has to be the normaliser of $G$ in $\mathop{\rm O}_n(\mathbb{R})$, and so does $\langle G,h'\rangle$. So they are not just conjugate, but equal. The set $H$ is the other coset of $G$ in its normaliser.

Here's why: Since $\langle G,h\rangle$ is finite, so is $\langle G,G^h\rangle$. This is then a finite subgroup of $\mathop{\rm SO}_n(\mathbb{R})$ containing $G$, and therefore by maximality we have $G^h=G$, so $h$ normalises $G$.