It is true if $X$ is normal and $Y$ is irreducible: Let $S:=X//G$. Since $X$ is affine and $G$ is reductive, the morphism $X\to S$ is a principal $G$-bundle (consequence of Luna's slice theorem, see Luna's original paper). This implies that also the geometric quotient $T:=X/H$ exists and $X\to T$ is a principal $H$-bundle. Since $Y\subseteq X$ is closed and $H$-stable, the geometric quotient $U:=Y/H$ exists and is a closed subvariety of $T$. Your assumptions imply that $Y$ intersects each $G$-orbit in an $H$-orbit. Thus $f:U\to S$ is bijective. Because $X$ is normal, also $S$ is normal. The irreducibility of $Y$ now implies that $f$ is an isomorphism, as claimed. This is a consequence of Zariski's main theorem: $f$ is birational because of $char=0$. Let $U\hookrightarrow \tilde S\to S$ be the factorization ${\rm finite}\circ{\rm open\ embedding}$. Then $\tilde S\to S$ is an isomorphism because $S$ is normal and $\tilde S$ is irreducible. Thus $f$ is a bijective open embedding, hence an isomorphism.
Remark: The irreducibility of $Y$ is essential: Take $G={\bf G}_m$, $X:={\bf A}^1\times{\bf G}_m$ with $G$ acting trivially on the first factor, and $H:=1$. Now put $Y:=\{xt=1\}\cup\{(0,1)\}$. Then $Y\to X/G={\bf A}^1$ is bijective but not an isomorphism.