Let $G$ be a smooth affine group scheme over a base $S$. $G$ acts on a scheme $X$ over $S$. Let $x$ be an $S$point in $X$. Then we have an orbit map $G\to X$. I wonder when the image (settheoretically) of this map is locally closed, and the induced scheme structure (the minimal one) on the orbit is smooth over $S$.

1$\begingroup$ You probably don't want the minimal (reduced) scheme structure if $S$ is not reduced (e.g. $G$ is trivial and the orbit is just the $S$ point $x$). Maybe it's better to use the scheme structure induced by the scheme theoretic image (if your orbit is open inside the closure of the set theoretic image). $\endgroup$– afhNov 23, 2021 at 20:45

$\begingroup$ @afh, i do want the orbit is open in the closure which is the schematic theoretic image. $\endgroup$– JJHNov 23, 2021 at 21:18
2 Answers
$\textbf{Edit by afh:}$ Unfortunately this answer is not correct. I apologize, there is a small bug in one of the last steps in the argument below (the surjective morphism of flat schemes at the end does not need to be a closed immersion/isomorphism). In fact the statement of the proposition below is not true even if the base is a DVR and $G$ is etale and quasifinite. The proof below only shows that the set theoretic image is locally closed when the base is a DVR.
Let me add another answer addressing the first comment of the OP above, where he asks for possible hypothesis that ensure the orbit is locally closed. I hope I am not making a lot of mistakes. In summary, there is a positive answer to the question if one assumes that the scheme theoretic stabilizer $G_x$ of the section $x$ is flat.
Here is the setup: let $S$ be a Noetherian scheme (probably can be removed) and let $X$ be a scheme of finite type over $S$. Let $G$ be a smooth affine algebraic group over $S$, and fix an action of $G$ on $X$. Let $x: S \to X$ be a section. If the scheme theoretic stabilizer $G_x$ is flat over $S$, then we can form the quotient algebraic space $G /G_x$ over $S$. By construction, the action morphism $$ G \to X, \; \; g \mapsto g \cdot x$$ factors through the quotient $G \to G/G_x$. Therefore we get a natural morphism $G/G_x \to X$. We shall denote $O= G/G_x$ and call it the orbit.
$\textbf{Proposition}$ In the setup above (under the assumption that $G_x$ is $S$flat), the orbit $O$ is represented by a smooth scheme over $S$ and the natural morphism $O \to X$ is a locally closed immersion.
Proof: First, since smoothness can be checked flat locally and $G$ is smooth, the fppf quotient morphism $G \to G/G_x$ shows that $O$ is smooth over $S$. It is not difficult to show that the orbit morphism $O \to X$ is a monomorphism of algebraic spaces (cf. Section 2.1 in https://arxiv.org/abs/0804.2242). Now, since $X$ is a scheme, we can apply https://stacks.math.columbia.edu/tag/03XX to the monomorphism $O \hookrightarrow X$ to conclude that $O$ is a scheme. It remains to check that $O \hookrightarrow X$ is a locally closed immersion.
We will use EGA IV (15.7.6). This Corollary in EGA says that if the valuative criterion for local properness (to be explained below) is satisfied for $ \phi: O \hookrightarrow X$, then the morphism $\phi$ factors as a composition $h \circ g$, where $h$ is an open immersion and $g$ is proper. In this case this would mean that $g$ is a proper monomorphism, and hence a closed immersion. In this (quasicompact) situation, this would in turn imply that $\phi: O \to X$ is a locally closed immersion. We are left to prove the valuative criteria mentioned above.
This is what we have to show. Let $R$ be a DVR with field of fractions $K$. Suppose that we are given a morphism $Spec(R) \to X$ that factors set theoretically through the set theoretic image $\phi(O)$. The local valuative criterion stipulates that any section $Spec(K) \to O \times_{X} Spec(K)$ must extend uniquely to a section $Spec(R) \to O \times_{X} Spec(R)$. In order to check this, we are free to base change using the morphism $Spec(R) \to X \to S$ in order to assume that the base $S$ is the spectrum of a DVR.
So we assume that $S$ is $Spec(R)$ with generic point $\eta$ and special point $s$. Take the scheme theoretic image of $Z \subset X$ of the morphism $\phi: O \hookrightarrow X$. Since $S$ is a DVR and $O$ is $S$flat, it follows that $Z$ is automatically flat over $X$ (this is the crucial reason why we passed to a DVR). It can be checked that $G$ still acts on $Z$ (as the scheme theoretic image of a quasicompact $G$equivariant morphism), so we might as well replace $X$ with $Z$ and assume that $X$ is flat and $O$ is scheme theoretically dense in $X$. Now taking scheme closure commutes with flat base change, so the generic fiber $O_{\eta} \hookrightarrow X_{\eta}$ is scheme theoretically dense. The usual argument for orbits over fields (notice that the construction of $O$ commutes with arbitrary basechange!) shows that $O_{\eta} \hookrightarrow X_{\eta}$ is an open immersion. Since $O$ is smooth, this shows that $X_{\eta}$ is geometrically reduced with the same dimension as $O_{\eta}$, and the boundary $B_{\eta} = X_{\eta} \setminus O_{\eta}$ has strictly smaller dimension. By flatness, the dimension of $X_{s}$ is the same as the dimension of $X_{\eta}$, and so we must have that $X_{s}$ and $O_{s}$ have the same dimension. Again, the usual argument for fields implies that $O_{s} \hookrightarrow X_s$ is locally closed, and since it is full dimension and smooth this means that the image of $O_s \subset X_s$ is open (but note that $X_s$ could be nonreduced, so we don't know yet that $O_s \to X_s$ is an open immersion).
We equip the boundary with $B_{\eta}$ with its reduced subscheme structure. Since $G_{\eta}$ is geometrically reduced, it acts on $B_{\eta}$. Take the scheme theoretic closure in $X$ of the boundary $B_{\eta} \to X_{\eta}$, let's call it $B$. $B$ is $G$stable, and the fibers have smaller dimension than the fibers of $O$. This means that $B_s$ must be disjoint from the open orbit $O_s$ of bigger dimension. Hence the closed subset $B$ is disjoint from the image of $\phi: O \hookrightarrow X$, and we can replace $X$ with $X \setminus B$. Hence we can assume that the generic fiber $\phi_{\eta} : O_{\eta} \to X_{\eta}$ is an isomorphism. Finally, by removing the closed complement $X_s \setminus O_s$ inside the generic fiber $X_s$, we can assume that the morphism of $R$flat schemes $\phi: O \to X$ becomes an infinitesimal closed immersion when restricted to the special fiber ($O_s$ will be the reduced subscheme of $X_s$ if $X_s$ is not reduced). Since the restriction to the generic fiber is also a closed immersion (isomorphism actually), this implies that $O \hookrightarrow X$ is a closed immersion. Since $O$ is schematically dense inside $X$, this means that $O \hookrightarrow X$ must be an isomorphism. In summary, we have shown that $O \to X$ induces an open immersion into its scheme theoretic closure. This implies that $O \to X$ is a locally closed immersion, and so by the same proposition in EGA IV (15.7.6), it satisfies the valuative criterion for local properness. QED
$\textbf{Last Remark/ Warning:}$ The scheme theoretic image $Z$ of the locally closed immersion $O \hookrightarrow X$ does not need to commute with passing to fibers. In other words, the fibers of the orbit $O$ do not need to be dense in the fibers of the closure $Z$. Therefore one cannot in principle conclude properties of the fibers of $Z$ in terms of properties of $O$. An example that I like is to let $R$ be a DVR with uniformizer $\pi$ and consider the scheme $X = Spec(R[s,t]/(st  \pi))$. We can let $\mathbb{G}_m$ act on $X$ with weight $1$ on $t$ and weight $1$ on $s$. Consider the section $(t,s) = (1,\pi)$. The stabilizer of this section is trivial, and the orbit $O$ is the open immersion $O \hookrightarrow X$ with closed complement the vanishing locus of $t$. At the generic fiber we have an isomorphism $O_{\eta} \cong X_{\eta}$ (there is a single orbit at the generic fiber), but at the special fiber the complement $X_s \setminus O_s$ is the vanishing locus of $t$, which contains two orbits: another open orbit of $(t,s) = (0,1)$, and the closed orbit at the origin $(t,s) = (0,0)$.
$\DeclareMathOperator\Spec{Spec}$Let $k$ be a field. Let $S= \mathbb{A}^1_k = \Spec(k[x])$ and $X= \mathbb{A}^2_k = \Spec(k[x,y])$.The constant additive group $\mathbb{G}_a\times S$ over $S$ acts on $X$ by the equation $$ t \cdot (x,y) = (x, y+xt).$$ The orbit of the $0$ section $\phi: S \to X$ defined by $$ \phi(x) = (x,0)$$ consists of the the union $V \cup (0,0)$, where $V$ is the open complement of the $y$axis ($x=0$), and $(0,0) \in \mathbb{A}^2$ is the origin. This can't possibly be flat over $S$, since the fiber over $x=0$ is $0$dimensional while the fiber over every other point is $1$dimensional.
I don't think that this image is locally closed either.

$\begingroup$ What if we assume the fiberwise orbit has the same dimension? Any reference on general theorem which guarantee local closedness and smoothness? Thanks. $\endgroup$– JJHNov 23, 2021 at 21:17

1$\begingroup$ just in case you know, my real question is: if BeilinsonDrinfeld Schubert cell is locally closed and smooth over the npower of the curve. When n=1, it is easy to see. What about n>2. $\endgroup$– JJHNov 23, 2021 at 22:09

$\begingroup$ That is an interesting question. I don't know the answer off the top of my head (you are saying that this cannot be extracted from the literature?). My first instinct (if I wanted to prove your question positively) would be to first equip the orbit with smooth structure and then try to show that it is a locally closed subscheme. $\endgroup$– afhNov 24, 2021 at 12:52

$\begingroup$ First, I would try to show that the stabilizer group scheme is smooth over the base (so you need to check flatness (a local condition), and then the fact that each fiber is smooth, which in char 0 is automatic and in this case it can be related to the local (affine grassmannian situation). Once you know that the stabilizer is smooth, you get that the quotient $O = G/Stab$ is a smooth algebraic space, and it admits a morphism $O \to Z$, where $Z$ is the scheme theoretic image of the morphism $G \to X$. Now you want to show that $O \to Z$ is represented by an open immersion. $\endgroup$– afhNov 24, 2021 at 12:55

$\begingroup$ You can just check that the morphism $O \to Z$ is flat, unramified and radicial. I believe all of these can be checked fiberwise (flatness by the fiberwise criterion for flatness). If you know the fibers of the scheme theoretic image $Z$ (is this what they call a spherical Schubert cell?), then you can again reduce to looking at a specific fiber, so you are working over a field + you can probably relate to the local situation (affine grassmannian). Sorry for the long comments, I don't know if this helps at all. Note that right away the problem in my example of $G_a$ is that Stab is not smooth. $\endgroup$– afhNov 24, 2021 at 13:00