I have a probably stupid question on schemes ... Let $S$ be a scheme, and let $A = \mathsf{Aut}(S)$ be its automorphism group. Does $A$ carry a scheme structure itself, that is, can one see $A$ as a group scheme ? Thanks !
The answer is yes when the scheme is flat and projective over the base. This follows from the existence of the Hom scheme, which in turn is proven via the existence of the Hilbert scheme.
A readable reference is Nitin Nitsure's part of the book Fundamental algebraic geometry. In particular Theorem 5.23 in his notes states that when S is noetherian, X projective and flat over S, and Y quasiprojective over S, then there exists a Homscheme parametrizing morphisms from X to Y over S. (The precise definition of the functor that this scheme represents is given in the text). The automorphism group scheme of X is then an open subscheme of $\mathrm{Hom}_S(X,X)$.
Addendum: The proof that Isom is open in Hom is very similar to the proof that Hom is open in Hilb. The map from the Hom scheme to $\mathrm{Hilb}_{X \times_S Y / S}$ is given by associating to a morphism $f : X \to Y$ its graph. Now the image consists of those $Z \subset X \times_S Y$ such that projection onto X induces an isomorphism $Z \cong X$, and the crucial part of the proof is showing that this is an open condition. (In Nitsure's notes this is Theorem 5.22.(b).) But then the condition that Z maps isomorphically onto Y is also open, and this is exactly the condition that defines the Isom scheme inside the Hom scheme.

1$\begingroup$ Forgive the naive question, but is there a nice explanation of why being an automorphism is an open condition? $\endgroup$ – Daniel Loughran Feb 11 '11 at 12:02

$\begingroup$ Thanks ! Is there a "direct" way to do it for flat and projective schemes, that is, without using the Hilbert scheme ? (For instance when one works with base a field.) $\endgroup$ – THC Feb 15 '11 at 9:28

$\begingroup$ Not that I know of. Maybe you can show that any automorphism extends to the ambient projective space (by cooking up an equivariant ample line bundle?), and then embed the automorphism group scheme in $\operatorname{Aut} \mathbf{P}^n = \mathrm{PGL}(n+1)$. P.S. Perhaps you already know this and I am talking down to you, but when you work over a field, the adjective flat can be omitted  every morphism to the spectrum of a field is flat. D.S. $\endgroup$ – Dan Petersen Feb 15 '11 at 11:59

$\begingroup$ Also, if you had asked Torsten or Sándor the same question, you would've gotten an authoritative answer. :p $\endgroup$ – Dan Petersen Feb 15 '11 at 12:04

$\begingroup$ Hi, Dan  yep, I knew (it's only "a for instance sentence") :) $\endgroup$ – THC Feb 15 '11 at 15:23
The automorphism group functor is not representable in general. Consider for instance the case of an algebraically closed field $k$ and $\mathbf A^2=\mathrm{Spec}\ k[x,y]$. Assume the automorphism group functor for this scheme is representable by an algebraic space $Aut$. For each $n$ we have an automorphism $(x,y)\mapsto(x+\sum_{1\leq i\leq n}t^iy^i,y)$ over $k[t]/(t^{n+1})$ giving a set of compatible morphisms $\mathrm{Spec}k[t]/(t^{n+1})\to Aut$. That system comes from a morphism $\mathrm{Spec}\ k[[t]]\to Aut$ (see Emerton's response to this question, which works also for maps into algebraic spaces at least as $k$ is algebraically closed). However, the corresponding automorphism would have to be $(x,y)\mapsto(x+\sum_{1\leq i}t^iy^i,y)$ which doesn't make sense.
Let $X\to B$ and $Y\to B$ be two flat, projective $B$schemes ($S$ is already taken ;). Then
Let $\mathscr Hom_B(X,Y)$ be the functor defined by $$\mathscr Hom_B(X,Y)(Z)=\{Z{\rm morphisms }\ X\times_B Z\to Y\times_B Z\}.$$ where $Z\to B$ is also a $B$scheme. Then $\mathscr Hom_B(X,Y)$ is represented by an open $B$subscheme $${\rm Hom}_B(X,Y)\subset {\rm Hilb}_B(X\times_BY).$$
The $\mathscr Hom$ functor has a subfunctor $\mathscr Isom$ consisting of those morphisms that define a relative isomorphism. This is represented by an open subscheme $${\rm Isom}_B(X,Y)\subset {\rm Hom}_B(X,Y).$$
Now if $B$ is a point, $X=Y$, then this $\rm Isom$ scheme can be identified with the automorphism group of $X$.

$\begingroup$ maybe you meant Zmorphisms $X\times_B Z \to Y\times_B Z$ in the definition of the functor? $\endgroup$ – Mattia Talpo Feb 10 '11 at 17:12
For projective, nonuniruled varities, the birational automorphism group can also be given a scheme structure. The reference is "Hanamura: Structure of birational automorphism groups, I: nonuniruled varieties". As one expects, the automorphism group will be a subscheme of that.
This is a partial answer to THC's question about a "direct" way to do this.
Let $X$ be a smooth projective variety such that $\omega_X$ is ample. Then there is some $m\in\mathbb N$ such that $\mathscr L=\omega_X^{\otimes m}$ is very ample, so $X$ has an embedding into $\mathbb P^N=\mathbb P(H^0(X,\mathscr L))$. Obviously $\mathscr L$ is invariant under any automorphism, and hence so is $H^0(X,\mathscr L)$. This implies that the automorphisms of $X$ extend to automorphisms of the ambient $\mathbb P^N$. In other words ${\rm Aut}(X)$ can be identified with a quotient of a subgroup of ${\rm Aut}(\mathbb P^N)$ and hence it inherits a natural scheme structure.
On the other hand it actually also follows that in this case ${\rm Aut}(X)$ is finite (see below why) and in fact it seems to me that this argument could only possibly work if something close to that is true.
Assume that $X$ is embedded into a projective space and all automorphisms of $X$ are induced by automorphisms of the ambient projective space. Then ${\rm Aut}(X)$ has a scheme structure (as above) and let ${\rm Aut}^\circ (X)$ denote the connected component of ${\rm Aut}(X)$ containing the identity. Then the quotient ${\rm Aut}(X)/{\rm Aut}^\circ (X)$ is a discrete group which is the quotient group of a linear algebraic group and hence it is finite.
The action of ${\rm Aut}^\circ (X)$ (in this case) exhibits $X$ as a uniruled variety and hence if $X$ is not uniruled (e.g., when $\omega_X$ is ample) it implies that ${\rm Aut}(X)$ is finite.
In other words if the scheme structure on ${\rm Aut}(X)$ given by the Hilbert scheme argument has infinitely many components, then it cannot be the quotient of a subgroup of ${\rm Aut}(\mathbb P^N)$. This happens for instance for K3 surfaces with an infinite automorphism group, in particular for any supersingular K3.

$\begingroup$ Of course, Sándor's answer explains much more, but it's also worth pointing out that on an elliptic curve $E$, there is no $\mathrm{Aut} E$invariant line bundle. Hence for every embedding of $E$ into projective space, almost all automorphisms do not extend to projective space. $\endgroup$ – Arend Bayer Feb 20 '11 at 12:14

$\begingroup$ Dear Sandor, very informative ! Can you say a word about "and hence it inherits a natural scheme structure" ? (Why this is when dealing with a subgroup quotient ?) I think the finiteness outcome is not an obstruction for me, since eventually my automorphism schemes will look like Chevalley group schemes ... $\endgroup$ – THC Feb 21 '11 at 12:46

$\begingroup$ How does the action of $Aut^∘(X)$ (in this case) exhibits $X$ as a uniruled variety ? $\endgroup$ – sawdada Nov 18 '17 at 15:52