Answer was edited
Here is a characterization I came up with:
Definition
- Let $G=Aut(M)$ (equiv. $G$ is a closed subgroup of $S_\infty$) and $N$ is a subgroup of $G$. Define $x\sim_N y$ iff there exists an automorphism $g\in N$ such that $g.x=y$.
- $G$ respects $\sim_N$ if for all $x,y$, $$x\sim_N y\text{ iff } g.x\sim_N g.y$$
Lemma Let $M$ be a countable model, $G=Aut(M)$ and $N$ be a closed subgroup of $G$. The following are equivalent:
- $N\vartriangleleft G$
- $G$ respects $\sim_N$
Proof. Unfold the definitions
\begin{align*}
N & \vartriangleleft G \text{ iff }\\
% \forall g\in G\; \forall n\in N\; \exists n'\in N,\; & g^{-1}ng=n' \text{ iff }\\
\forall x\in M \forall g\in G\; \forall n\in N\; \exists n'\in N,\; &
g^{-1}ng.x=n'.x \text{ iff }\\
\forall x,y \in M \forall g\in G,\; & (\exists n\in N,\; y=g^{-1}ng.x)
\leftrightarrow (\exists n'\in N,\; y=n'.x) \text{ iff }\\
\forall x,y \in M \forall g\in G,\; & (\exists n\in N,\; g.y=ng.x)
\leftrightarrow (\exists n'\in N,\;y=n'.x) \text{ iff }\\
\forall x,y \in M \forall g\in G,\; &
(g.x\sim_N g.y) \leftrightarrow (x\sim_N y)
\end{align*}
End of Proof.
Stabilizers
As a special case we prove that if $X$ is a $G$-invariant set, then point-wise stabilizer of $X$ is a normal subgroup of $G$.
Definition If $G$ is a group acting on $\mathbb{N}$ and $n\in\mathbb{N}$, then $G_n$ denotes the stabilizer of $n$ in $G$. If $X$ is a subset of $\mathbb{N}$, $G_{X}$ denotes the pointwise-stabilizer of $X$.
Lemma 1 For all $n,g$, $G_{g(n)}= g G_n g^{-1}$.
Proof. $f\in G_{g(n)}$ iff $f(g(n))=g(n)$ iff $g^{-1}(f(g(n)))=n$ iff $g^{-1}(f(g))\in G_n$ iff $f\in g G_n g^{-1}$. $\square$
Corollary 2 For $X\subset\mathbb{N}$, $G_{g(X)}=g G_X g^{-1}$.
Lemma 3 Let $G=Aut(M)$, for some $M$, and let $X$ be a $G$-invariant set. Then the pointwise-stabilizer of $X$ is a normal subgroup in $G$.
Proof. Let $g\in G$. Since $X$ is G-invariant, $g(X)=X$. By Corollary 2, $G_X=G_{g(X)}=g G_X g^{-1}$, which proves the result. $\square$.
Some Comments
- By Lemma 3, the number of normal subgroups of $G$ correlates with the number of $G$-orbits. We can construct an example of $G$ with $\aleph_0$-many normal subgroups, even with $2^{\aleph_0}$-many normal subgroups.
- In view of Lemma 3, $Aut(M)$ is a simple group, only if $Aut(M)$ acts trasitively on $M$, i.e. has only one orbit. So, it is necessary in the result mentioned by Igor Rivin that the $Aut(M)$ acts transitively on $M$.
- By a theorem of Dana Scott, every $Aut(M)$-orbit is defined by an $L_{\omega_1,\omega}$-sentence (in the same vocabulary as $M$). If $N$ is a normal subgroup of $Aut(M)$, what can we say about the action of $N$ on $M$?