I don't think that this is true without extra hypotheses like normality of $X$ (if $X$ is normal then it follows as explained by Damian Rossler in the comments). Here is a possible example.

We will take $G = GL_2$ and $H=1$ (the latter is just the trivial group). Let $C$ be the cuspidal curve $C = Spec(k[x,y]/(y^2-x^3))$.

Let's take $X = GL_2 \times C$, equipped with the natural action of $GL_2$ by right multiplication on the first factor (this is just the trivial $GL_2$-bundle over $C$).

Inside $GL_2$ we have the additive group $\mathbb{G}_a$ of strictly upper triangular matrices. So we have a closed subvariety $Z = \mathbb{G}_a \times C$ inside $X$. We can write $Z = Spec(k[x,y,t](y^2-x^3))$. Inside $Z$ we can define $Y$ to be the vanishing locus of the polynomial $xt-y$. So we get a closed subvariety $Y \hookrightarrow X$ given by

$$ Y = Spec(k[x,y,t]/(y^2-x^3, xt-y))$$

Notice that $X/GL_2 =C$ and the projection $Y \to C$ onto the second component is the map $Y/H \to X/GL_2$ in this case.

$Y$ is actually isomorphic to $\mathbb{A}^1$, and the projection $Y \to C$ exhibits $Y$ as the normalization of the cuspidal curve $C$. In particular $Y \to C$ is not an isomorphism.

Note that the projection $Y \to C$ is bijective on points, so $Y$ contains exactly one point in each fiber of the $GL_2$-bundle $X \to C$. This implies that $GL_2 \cdot Y = X$ and also that there are no two closed points of $Y$ that belong to the same $GL_2$-orbit. In other words, the hypotheses are satisfied for $Y \hookrightarrow X$ (with $H = 1$) and the map $Y/H \to X/G$ is not an isomorphism.

One can probably do variations of this to get examples with $H$ nontrivial.