Hi there,
I have a probably stupid question on schemes ... Let S be a scheme, and let A be its automorphism group. Does A carry a scheme structure itself, that is, can one see A as a group scheme ? Thanks !
Hi there, I have a probably stupid question on schemes ... Let S be a scheme, and let A be its automorphism group. Does A carry a scheme structure itself, that is, can one see A as a group scheme ? Thanks ! 


The answer is yes when the scheme is flat and projective over the base. This follows from the existence of the Hom scheme, which in turn is proven via the existence of the Hilbert scheme. A readable reference is Nitin Nitsure's part of the book Fundamental algebraic geometry. In particular Theorem 5.23 in his notes states that when S is noetherian, X projective and flat over S, and Y quasiprojective over S, then there exists a Homscheme parametrizing morphisms from X to Y over S. (The precise definition of the functor that this scheme represents is given in the text). The automorphism group scheme of X is then an open subscheme of $\mathrm{Hom}_S(X,X)$. Addendum: The proof that Isom is open in Hom is very similar to the proof that Hom is open in Hilb. The map from the Hom scheme to $\mathrm{Hilb}_{X \times_S Y / S}$ is given by associating to a morphism $f : X \to Y$ its graph. Now the image consists of those $Z \subset X \times_S Y$ such that projection onto X induces an isomorphism $Z \cong X$, and the crucial part of the proof is showing that this is an open condition. (In Nitsure's notes this is Theorem 5.22.(b).) But then the condition that Z maps isomorphically onto Y is also open, and this is exactly the condition that defines the Isom scheme inside the Hom scheme. 


The automorphism group functor is not representable in general. Consider for instance the case of an algebraically closed field $k$ and $\mathbf A^2=\mathrm{Spec}\ k[x,y]$. Assume the automorphism group functor for this scheme is representable by an algebraic space $Aut$. For each $n$ we have an automorphism $(x,y)\mapsto(x+\sum_{1\leq i\leq n}t^iy^i,y)$ over $k[t]/(t^{n+1})$ giving a set of compatible morphisms $\mathrm{Spec}k[t]/(t^{n+1})\to Aut$. That system comes from a morphism $\mathrm{Spec}\ k[[t]]\to Aut$ (see Emerton's response to this question, which works also for maps into algebraic spaces at least as $k$ is algebraically closed). However, the corresponding automorphism would have to be $(x,y)\mapsto(x+\sum_{1\leq i}t^iy^i,y)$ which doesn't make sense. 


Let $X\to B$ and $Y\to B$ be two flat, projective $B$schemes ($S$ is already taken ;). Then Let $\mathscr Hom_B(X,Y)$ be the functor defined by $$\mathscr Hom_B(X,Y)(Z)=\{Z{\rm morphisms }\ X\times_B Z\to Y\times_B Z\}.$$ where $Z\to B$ is also a $B$scheme. Then $\mathscr Hom_B(X,Y)$ is represented by an open $B$subscheme $${\rm Hom}_B(X,Y)\subset {\rm Hilb}_B(X\times_BY).$$ The $\mathscr Hom$ functor has a subfunctor $\mathscr Isom$ consisting of those morphisms that define a relative isomorphism. This is represented by an open subscheme $${\rm Isom}_B(X,Y)\subset {\rm Hom}_B(X,Y).$$ Now if $B$ is a point, $X=Y$, then this $\rm Isom$ scheme can be identified with the automorphism group of $X$. 


For projective, nonuniruled varities, the birational automorphism group can also be given a scheme structure. The reference is "Hanamura: Structure of birational automorphism groups, I: nonuniruled varieties". As one expects, the automorphism group will be a subscheme of that. 


This is a partial answer to THC's question about a "direct" way to do this. Let $X$ be a smooth projective variety such that $\omega_X$ is ample. Then there is some $m\in\mathbb N$ such that $\mathscr L=\omega_X^{\otimes m}$ is very ample, so $X$ has an embedding into $\mathbb P^N=\mathbb P(H^0(X,\mathscr L))$. Obviously $\mathscr L$ is invariant under any automorphism, and hence so is $H^0(X,\mathscr L)$. This implies that the automorphisms of $X$ extend to automorphisms of the ambient $\mathbb P^N$. In other words ${\rm Aut}(X)$ can be identified with a quotient of a subgroup of ${\rm Aut}(\mathbb P^N)$ and hence it inherits a natural scheme structure. On the other hand it actually also follows that in this case ${\rm Aut}(X)$ is finite (see below why) and in fact it seems to me that this argument could only possibly work if something close to that is true. Assume that $X$ is embedded into a projective space and all automorphisms of $X$ are induced by automorphisms of the ambient projective space. Then ${\rm Aut}(X)$ has a scheme structure (as above) and let ${\rm Aut}^\circ (X)$ denote the connected component of ${\rm Aut}(X)$ containing the identity. Then the quotient ${\rm Aut}(X)/{\rm Aut}^\circ (X)$ is a discrete group which is the quotient group of a linear algebraic group and hence it is finite. The action of ${\rm Aut}^\circ (X)$ (in this case) exhibits $X$ as a uniruled variety and hence if $X$ is not uniruled (e.g., when $\omega_X$ is ample) it implies that ${\rm Aut}(X)$ is finite. In other words if the scheme structure on ${\rm Aut}(X)$ given by the Hilbert scheme argument has infinitely many components, then it cannot be the quotient of a subgroup of ${\rm Aut}(\mathbb P^N)$. This happens for instance for K3 surfaces with an infinite automorphism group, in particular for any supersingular K3. 

