Smoothness of orbit of group scheme

Question

Let $G$ be a smooth affine group scheme over a base $S$. $G$ acts on a scheme $X$ over $S$. Let $x$ be an $S$-point in $X$. Then we have an orbit map $G\to X$. I wonder when the image (set-theoretically) of this map is locally closed, and the induced scheme structure (the minimal one) on the orbit is smooth over $S$.

Answer 1

$\textbf{Edit by afh:}$ Unfortunately this answer is not correct. I apologize, there is a small bug in one of the last steps in the argument below (the surjective morphism of flat schemes at the end does not need to be a closed immersion/isomorphism). In fact the statement of the proposition below is not true even if the base is a DVR and $G$ is etale and quasifinite. The proof below only shows that the set theoretic image is locally closed when the base is a DVR.

Let me add another answer addressing the first comment of the OP above, where he asks for possible hypothesis that ensure the orbit is locally closed. I hope I am not making a lot of mistakes. In summary, there is a positive answer to the question if one assumes that the scheme theoretic stabilizer $G_x$ of the section $x$ is flat.

Here is the setup: let $S$ be a Noetherian scheme (probably can be removed) and let $X$ be a scheme of finite type over $S$. Let $G$ be a smooth affine algebraic group over $S$, and fix an action of $G$ on $X$. Let $x: S \to X$ be a section. If the scheme theoretic stabilizer $G_x$ is flat over $S$, then we can form the quotient algebraic space $G /G_x$ over $S$. By construction, the action morphism $$ G \to X, \; \; g \mapsto g \cdot x$$ factors through the quotient $G \to G/G_x$. Therefore we get a natural morphism $G/G_x \to X$. We shall denote $O= G/G_x$ and call it the orbit.

$\textbf{Proposition}$ In the setup above (under the assumption that $G_x$ is $S$-flat), the orbit $O$ is represented by a smooth scheme over $S$ and the natural morphism $O \to X$ is a locally closed immersion.

Proof: First, since smoothness can be checked flat locally and $G$ is smooth, the fppf quotient morphism $G \to G/G_x$ shows that $O$ is smooth over $S$. It is not difficult to show that the orbit morphism $O \to X$ is a monomorphism of algebraic spaces (cf. Section 2.1 in https://arxiv.org/abs/0804.2242). Now, since $X$ is a scheme, we can apply https://stacks.math.columbia.edu/tag/03XX to the monomorphism $O \hookrightarrow X$ to conclude that $O$ is a scheme. It remains to check that $O \hookrightarrow X$ is a locally closed immersion.

We will use EGA IV (15.7.6). This Corollary in EGA says that if the valuative criterion for local properness (to be explained below) is satisfied for $ \phi: O \hookrightarrow X$, then the morphism $\phi$ factors as a composition $h \circ g$, where $h$ is an open immersion and $g$ is proper. In this case this would mean that $g$ is a proper monomorphism, and hence a closed immersion. In this (quasicompact) situation, this would in turn imply that $\phi: O \to X$ is a locally closed immersion. We are left to prove the valuative criteria mentioned above.

This is what we have to show. Let $R$ be a DVR with field of fractions $K$. Suppose that we are given a morphism $Spec(R) \to X$ that factors set theoretically through the set theoretic image $\phi(O)$. The local valuative criterion stipulates that any section $Spec(K) \to O \times_{X} Spec(K)$ must extend uniquely to a section $Spec(R) \to O \times_{X} Spec(R)$. In order to check this, we are free to base change using the morphism $Spec(R) \to X \to S$ in order to assume that the base $S$ is the spectrum of a DVR.

So we assume that $S$ is $Spec(R)$ with generic point $\eta$ and special point $s$. Take the scheme theoretic image of $Z \subset X$ of the morphism $\phi: O \hookrightarrow X$. Since $S$ is a DVR and $O$ is $S$-flat, it follows that $Z$ is automatically flat over $X$ (this is the crucial reason why we passed to a DVR). It can be checked that $G$ still acts on $Z$ (as the scheme theoretic image of a quasicompact $G$-equivariant morphism), so we might as well replace $X$ with $Z$ and assume that $X$ is flat and $O$ is scheme theoretically dense in $X$. Now taking scheme closure commutes with flat base change, so the generic fiber $O_{\eta} \hookrightarrow X_{\eta}$ is scheme theoretically dense. The usual argument for orbits over fields (notice that the construction of $O$ commutes with arbitrary base-change!) shows that $O_{\eta} \hookrightarrow X_{\eta}$ is an open immersion. Since $O$ is smooth, this shows that $X_{\eta}$ is geometrically reduced with the same dimension as $O_{\eta}$, and the boundary $B_{\eta} = X_{\eta} \setminus O_{\eta}$ has strictly smaller dimension. By flatness, the dimension of $X_{s}$ is the same as the dimension of $X_{\eta}$, and so we must have that $X_{s}$ and $O_{s}$ have the same dimension. Again, the usual argument for fields implies that $O_{s} \hookrightarrow X_s$ is locally closed, and since it is full dimension and smooth this means that the image of $O_s \subset X_s$ is open (but note that $X_s$ could be nonreduced, so we don't know yet that $O_s \to X_s$ is an open immersion).

We equip the boundary with $B_{\eta}$ with its reduced subscheme structure. Since $G_{\eta}$ is geometrically reduced, it acts on $B_{\eta}$. Take the scheme theoretic closure in $X$ of the boundary $B_{\eta} \to X_{\eta}$, let's call it $B$. $B$ is $G$-stable, and the fibers have smaller dimension than the fibers of $O$. This means that $B_s$ must be disjoint from the open orbit $O_s$ of bigger dimension. Hence the closed subset $B$ is disjoint from the image of $\phi: O \hookrightarrow X$, and we can replace $X$ with $X \setminus B$. Hence we can assume that the generic fiber $\phi_{\eta} : O_{\eta} \to X_{\eta}$ is an isomorphism. Finally, by removing the closed complement $X_s \setminus O_s$ inside the generic fiber $X_s$, we can assume that the morphism of $R$-flat schemes $\phi: O \to X$ becomes an infinitesimal closed immersion when restricted to the special fiber ($O_s$ will be the reduced subscheme of $X_s$ if $X_s$ is not reduced). Since the restriction to the generic fiber is also a closed immersion (isomorphism actually), this implies that $O \hookrightarrow X$ is a closed immersion. Since $O$ is schematically dense inside $X$, this means that $O \hookrightarrow X$ must be an isomorphism. In summary, we have shown that $O \to X$ induces an open immersion into its scheme theoretic closure. This implies that $O \to X$ is a locally closed immersion, and so by the same proposition in EGA IV (15.7.6), it satisfies the valuative criterion for local properness. QED

$\textbf{Last Remark/ Warning:}$ The scheme theoretic image $Z$ of the locally closed immersion $O \hookrightarrow X$ does not need to commute with passing to fibers. In other words, the fibers of the orbit $O$ do not need to be dense in the fibers of the closure $Z$. Therefore one cannot in principle conclude properties of the fibers of $Z$ in terms of properties of $O$. An example that I like is to let $R$ be a DVR with uniformizer $\pi$ and consider the scheme $X = Spec(R[s,t]/(st - \pi))$. We can let $\mathbb{G}_m$ act on $X$ with weight $-1$ on $t$ and weight $1$ on $s$. Consider the section $(t,s) = (1,\pi)$. The stabilizer of this section is trivial, and the orbit $O$ is the open immersion $O \hookrightarrow X$ with closed complement the vanishing locus of $t$. At the generic fiber we have an isomorphism $O_{\eta} \cong X_{\eta}$ (there is a single orbit at the generic fiber), but at the special fiber the complement $X_s \setminus O_s$ is the vanishing locus of $t$, which contains two orbits: another open orbit of $(t,s) = (0,1)$, and the closed orbit at the origin $(t,s) = (0,0)$.

Answer 2

$\DeclareMathOperator\Spec{Spec}$Let $k$ be a field. Let $S= \mathbb{A}^1_k = \Spec(k[x])$ and $X= \mathbb{A}^2_k = \Spec(k[x,y])$.The constant additive group $\mathbb{G}_a\times S$ over $S$ acts on $X$ by the equation $$ t \cdot (x,y) = (x, y+xt).$$ The orbit of the $0$ section $\phi: S \to X$ defined by $$ \phi(x) = (x,0)$$ consists of the the union $V \cup (0,0)$, where $V$ is the open complement of the $y$-axis ($x=0$), and $(0,0) \in \mathbb{A}^2$ is the origin. This can't possibly be flat over $S$, since the fiber over $x=0$ is $0$-dimensional while the fiber over every other point is $1$-dimensional.

I don't think that this image is locally closed either.