Now that you've added the maximality assumption, $\langle G,h\rangle$ has to be the normaliser of $G$ in $\mathop{\rm O}_n(\mathbb{R})$, and so does $\langle G,h'\rangle$. So they are not just conjugate, but equal. The set $H$ is the other coset of $G$ in its normaliser.

Here's why: Since $\langle G,h\rangle$ is finite, so is $\langle G,G^h\rangle$. This is then a finite subgroup of $\mathop{\rm SO}_n(\mathbb{R})$ containing $G$, and therefore by maximality we have $G^h=G$, so $h$ normalises $G$. 

To expand the argument a bit, the centraliser of $G$ in $\mathop{\rm SO}_n(\mathbb{R})$ is $Z(G)$, since otherwise $G$ can be enlarged to a larger finite subgroup. So the centraliser in $\mathop{\rm O}_n(\mathbb{R})$ is also finite. The normaliser mod the centraliser is a group of automorphisms of a finite group, so is finite. Therefore the normaliser is finite.