Skip to main content
2 of 2
added 470 characters in body
Steve Costenoble
  • 2.1k
  • 1
  • 13
  • 22

Your definition implies that $$ \tilde S = \bigcup_{p\in M} T_pM^{G_p}. $$ In particular, $\pi(\tilde S) = M$, and $\tilde S$ will be a submanifold of $TM$ iff the dimension of $T_pM^{G_p}$ is the same for all $p\in M$.

$G$ being connected won't necessarily make this happen: Another counterexample is $S^1$ acting by rotation on $S^2$, fixing the north and south poles, $n$ and $s$. $\tilde S$ is then $T(S^2\setminus\{n,s\})\cup \{n,s\}$, and the points $n$ and $s$ do not have neighborhoods homeomorphic to open balls.

EDIT

To address the modified question: If $p\in S$, then the orbit $Gp$ has a neighborhood homeomorphic to $G\times_{G_p} \mathcal H_p$ and we're assuming that $\mathcal H_p^{G_p} \neq 0$. If $q\in \mathcal H_p$ is any point, then $\mathcal H_p^{G_p}$ will be fixed by $G_q$ and still be orthogonal to $Gq$, hence $q$ and every point in $Gq$ will be in $S$. This shows that a neighborhood of $Gp$ is contained in $S$. Thus $S$ is an open submanifold of $M$.

Steve Costenoble
  • 2.1k
  • 1
  • 13
  • 22