Let $G$ be a connected linear algebraic group (say, over an algebraically closed field) and let $H < G$ be a closed connected subgroup. Let $N_G(H)$ be the normalizer of $H$ in $G$, and assume that $N_G(H)=H$. Does there necessarily exist a single element $h \in H$ such that if $g \in H$ satisfies $g h g^{-1} \in G$, then $g \in H$?

In the setting I am in $H$ is solvable, but I doubt that is relevant to the question.