Measure on orbits of $N$ under conjugation by $H$

Question

Let $G$ be a locally compact topological group with closed subgroups $H, N$ and $H$ normalizing $N$. Then $H$ acts continuously on $N$ by conjugation. If it will help, assume that $N$ is nilpotent, so that Haar measures on $N$ and all its closed subgroups are unimodular.

If $n \in N$, let $O = O(n)$ be the orbit of $n$ under the action of $H$:

$$O = \{ hnh^{-1} : h \in H \}$$

Then the group action $H\times O \rightarrow O$ is continuous, transitive, and for fixed $n' \in O$, the map $H \rightarrow O$, $h \mapsto hn'h^{-1}$ is open. This implies that $O$ is homeomorphic to the space of left cosets $H/\textrm{Stab } n$ in the quotient topology, where

$$\textrm{Stab } n = H \cap Z_G(n)= \{ h \in H : hnh^{-1} = n\}$$

Thus to talk about the existence of $H$-invariant measures on $O$, it suffices to talk about them on $H/H \cap Z_G(n)$. Standard results about the Haar measures on $H$ and $H \cap Z_G(n)$ give conditions on when such a measure exists.

On the other hand, suppose instead of considering just one orbit, we consider all of them. That is, we consider the space $H \setminus N$ of orbits of $N$ under the action of $H$. I suppose we give this space the quotient topology. Then is there some natural Borel measure on $H \setminus N$, perhaps related in some way to the Haar measures on $H$ and $N$?

Answer 1

One should take care here. While when considering algebraic group actions typically orbits are locally closed, in general this is not the case.

Consider for example the multiplication action of $H=\mathbb{Q}^*$ on $N=\mathbb{R}$ and form the corresponding semidirect product $G=H\ltimes N$. Topologize it by taking the standard topology on $N$ and the discrete topology on $H$. Then you have an $H$ invariant measure on each $H$-orbit in $N$ (the counting measure), but it doesn't have much to do with the Haar measure of $N$. The quotient topology on $N/H$ will be trivial, taking out one closed point corresponding to $0$.

Another example you should carry in mind is $H=\text{SL}_2(\mathbb{Z})$ acting on $H=\mathbb{R}^2/\mathbb{Z}^2$. Again take $G=H\ltimes N$. Now the quotient topology on $N/H$ has countable dense subset of closed points corresponding to rational points of $H$. Taking it out you are left again with the trivial topology. Here too, one cannot make any reasonable relation between measures on this space and the Haar measure of $N$. Note that in this example the Haar measure on $N$ is $H$ invariant, while in the first example it was not. You could not tell this by just looking on $N/H$.