Adam has given the simplest type of concrete example of what can go wrong, But since the formulation of the question has caused some confusion, it may be worth commenting at length on the broader algebraic group background involved. Here the given group $G$ can be any connected reductive algebraic group over an algebraically closed field $K$ (not just in characteristic 0), while $H$ is a (closed but not necessarily connected) subgroup. The only assumption is that $G$ acts (as an algebraic group) on some affine variety $X$ with a closed orbit $G \cdot x$. The question is whether the orbit $H \cdot x$ must also be closed. [Some other language involving maximal compact subgroups in the question is irrelevant.] The answer is no, though the approaches of Adam and Jason both look somewhat ad hoc.

In Adam's example, $G$ is taken to be the product of two copies of the general linear group $X=\mathrm{GL}(2,K)$. Since $X$ is a homogeneous space for its natural left and right multiplication actions (taking inverses on the right so that multiplication becomes a left action), the combined left action of $G$ on $X$ makes $X$ into a homogeneous space. It is the orbit of any invertible matrix $x$, for instance $I$ whose isotropy group is the diagonally embedded copy of $X$ in $G$ acting by conjugation on itself (other isotropy groups being conjugate to this one). In particular, $X$ is just the affine homogeneous space of $G$ modulo that reductive subgroup. This construction is quite general.

In the example, the point $x \in X$ is chosen to be the standard unipotent Jordan block (a regular unipotent element of the group $X$), while $H$ is taken to be the standard maximal torus in $\mathrm{SL}(2,K)$ inside the diagonally embedded copy of the group $X$ in $G$. Now $H$ is reductive but has a mixture of orbits on the variety $X$, a closed orbit being that of $I$. But the orbit of $x$ fails to be closed (in the Zariski topology, or the analytic topology when that's appropriate). However, this orbit is a perfectly good affine variety, being in fact the quotient of $H$ by a subgroup of order 2.

In general, closed orbits always exist for such algebraic group actions (e.g., orbits of smallest dimension). But for a semisimple group, an orbit under the conjugation action is closed precisely when it is the orbit of a semisimple element. Working with unipotent elements is a natural way to get non-closed orbits. For example, if we chose $H$ instead to be the full diagonally embedded copy of $X$ in $G$, the orbit of $x$ would be its non-closed conjugacy class in $X$ while the isotropy group (= centralizer) would in fact be non-reductive. For more background on homogeneous spaces, see the old papers:

R.W. Richardson, Affine coset spaces of reductive algebraic groups, Bull. London Math. Soc. 9 (1977), 38-41

E. Cline, B. Parshall, and L. Scott, Induced modules and affine quotients, Math. Ann. 230 (1977), 1-14

Adam has given the simplest type of concrete example of what can go wrong, But since the formulation of the question has caused some confusion, it may be worth commenting at length on the broader algebraic group background involved. Here the given group $G$ can be any connected reductive algebraic group over an algebraically closed field $K$ (not just in characteristic 0), while $H$ is a (closed but not necessarily connected) reductive subgroup. The only assumption is that $G$ acts (as an algebraic group) on some affine variety $X$ with a closed orbit $G \cdot x$. The question is whether the orbit $H \cdot x$ must also be closed. [Some other language involving maximal compact subgroups in the question is irrelevant.] The answer is no, though the approaches of Adam and Jason both look somewhat ad hoc.

In Adam's example, $G$ is taken to be the product of two copies of the general linear group $X=\mathrm{GL}(2,K)$. Since $X$ is a homogeneous space for its natural left and right multiplication actions (taking inverses on the right so that multiplication becomes a left action), the combined left action of $G$ on $X$ makes $X$ into a homogeneous space. It is the orbit of any invertible matrix $x$, for instance $I$ whose isotropy group is the diagonally embedded copy of $X$ in $G$ acting by conjugation on itself (other isotropy groups being conjugate to this one). In particular, $X$ is just the affine homogeneous space of $G$ modulo that reductive subgroup. This construction is quite general.

In the example, the point $x \in X$ is chosen to be the standard unipotent Jordan block (a regular unipotent element of the group $X$), while $H$ is taken to be the standard maximal torus in $\mathrm{SL}(2,K)$ inside the diagonally embedded copy of the group $X$ in $G$. Now $H$ is reductive but has a mixture of orbits on the variety $X$, a closed orbit being that of $I$. But the orbit of $x$ fails to be closed (in the Zariski topology, or the analytic topology when that's appropriate). However, this orbit is a perfectly good affine variety, being in fact the quotient of $H$ by a subgroup of order 2.

In general, closed orbits always exist for such algebraic group actions (e.g., orbits of smallest dimension). But for a semisimple group, an orbit under the conjugation action is closed precisely when it is the orbit of a semisimple element. Working with unipotent elements is a natural way to get non-closed orbits. For example, if we chose $H$ instead to be the full diagonally embedded copy of $X$ in $G$, the orbit of $x$ would be its non-closed conjugacy class in $X$ while the isotropy group (= centralizer) would in fact be non-reductive. For more background on homogeneous spaces, see the old papers:

R.W. Richardson, Affine coset spaces of reductive algebraic groups, Bull. London Math. Soc. 9 (1977), 38-41

E. Cline, B. Parshall, and L. Scott, Induced modules and affine quotients, Math. Ann. 230 (1977), 1-14

Adam has given the simplest type of concrete example of what can go wrong, But since the formulation of the question has caused some confusion, it may be worth commenting at length on the broader algebraic group background involved. Here the given group $G$ can be any connected reductive algebraic group over an algebraically closed field $K$ (not just in characteristic 0), while $H$ is a (closed but not necessarily connected) subgroup. The only assumption is that $G$ acts (as an algebraic group) on some affine variety $X$ with a closed orbit $G \cdot X$. The question is whether the orbit $H \cdot x$ must also be closed. [Some other language involving maximal compact subgroups in the question is irrelevant.] The answer is no, though the approaches of Adam and Jason both look somewhat ad hoc.

In Adam's example, $G$ is taken to be the product of two copies of the general linear group $X=\mathrm{GL}(2,K)$. Since $X$ is a homogeneous space for its natural left and right multiplication actions (taking inverses on the right so that multiplication becomes a left action), the combined left action of $G$ on $X$ makes $X$ into a homogeneous space. It is the orbit of any invertible matrix $x$, for instance $I$ whose isotropy group is the diagonally embedded copy of $X$ in $G$ acting by conjugation on itself (other isotropy groups being conjugate to this one). In particular, $X$ is just the affine homogeneous space of $G$ modulo that reductive subgroup. This construction is quite general.

In the example, the point $x \in X$ is chosen to be the standard unipotent Jordan block (a regular unipotent element of the group $X$), while $H$ is taken to be the diagonally embedded copy of the standard maximal torus in $\mathrm{SL}(2,K)$ inside the diagonally embedded copy of the group $X$ in $G$. Now $H$ is reductive but has a mixture of orbits on the variety $X$, a closed orbit being that of $I$. But the orbit of $x$ fails to be closed (in the Zariski topology, or the analytic topology when that's appropriate). However, this orbit is a perfectly good affine variety, being in fact the quotient of $H$ by a subgroup of order 2.

In general, closed orbits always exist for such algebraic group actions (e.g., orbits of smallest dimension). But for a semisimple group, an orbit under the conjugation action is closed precisely when it is the orbit of a semisimple element. Working with unipotent elements is a natural way to get non-closed orbits. For example, if we chose $H$ instead to be the full diagonally embedded copy of $X$ in $G$, the orbit of $x$ would be its non-closed conjugacy class in $X$ while the isotropy group (= centralizer) would in fact be non-reductive. For more background on homogeneous spaces, see the old papers:

R.W. Richardson, Affine coset spaces of reductive algebraic groups, Bull. London Math. Soc. 9 (1977), 38-41

E. Cline, B. Parshall, and L. Scott, Induced modules and affine quotients, Math. Ann. 230 (1977), 1-14

Adam has given the simplest type of concrete example of what can go wrong, But since the formulation of the question has caused some confusion, it may be worth commenting at length on the broader algebraic group background involved. Here the given group $G$ can be any connected reductive algebraic group over an algebraically closed field $K$ (not just in characteristic 0), while $H$ is a (closed but not necessarily connected) subgroup. The only assumption is that $G$ acts (as an algebraic group) on some affine variety $X$ with a closed orbit $G \cdot x$. The question is whether the orbit $H \cdot x$ must also be closed. [Some other language involving maximal compact subgroups in the question is irrelevant.] The answer is no, though the approaches of Adam and Jason both look somewhat ad hoc.

In Adam's example, $G$ is taken to be the product of two copies of the general linear group $X=\mathrm{GL}(2,K)$. Since $X$ is a homogeneous space for its natural left and right multiplication actions (taking inverses on the right so that multiplication becomes a left action), the combined left action of $G$ on $X$ makes $X$ into a homogeneous space. It is the orbit of any invertible matrix $x$, for instance $I$ whose isotropy group is the diagonally embedded copy of $X$ in $G$ acting by conjugation on itself (other isotropy groups being conjugate to this one). In particular, $X$ is just the affine homogeneous space of $G$ modulo that reductive subgroup. This construction is quite general.

In the example, the point $x \in X$ is chosen to be the standard unipotent Jordan block (a regular unipotent element of the group $X$), while $H$ is taken to be the standard maximal torus in $\mathrm{SL}(2,K)$ inside the diagonally embedded copy of the group $X$ in $G$. Now $H$ is reductive but has a mixture of orbits on the variety $X$, a closed orbit being that of $I$. But the orbit of $x$ fails to be closed (in the Zariski topology, or the analytic topology when that's appropriate). However, this orbit is a perfectly good affine variety, being in fact the quotient of $H$ by a subgroup of order 2.

In general, closed orbits always exist for such algebraic group actions (e.g., orbits of smallest dimension). But for a semisimple group, an orbit under the conjugation action is closed precisely when it is the orbit of a semisimple element. Working with unipotent elements is a natural way to get non-closed orbits. For example, if we chose $H$ instead to be the full diagonally embedded copy of $X$ in $G$, the orbit of $x$ would be its non-closed conjugacy class in $X$ while the isotropy group (= centralizer) would in fact be non-reductive. For more background on homogeneous spaces, see the old papers:

R.W. Richardson, Affine coset spaces of reductive algebraic groups, Bull. London Math. Soc. 9 (1977), 38-41

E. Cline, B. Parshall, and L. Scott, Induced modules and affine quotients, Math. Ann. 230 (1977), 1-14