I am not sure that all automorphism groups of algebraic varieties have natrual algebraic group structure. But if the automorphism group of a variety has algebraic group structure, how do I know the automorphism group is an algebraic group. For example, the automorphism group of an elliptic curve $A$ is an extension of the group $G$ of automorphisms which preserve the structure of the elliptic curve, by the group $A(k)$ of translations in the points of $A$, i.e. the sequence of groups $0\to A(k)\to \text{Aut}(A)\to G \to 0$ is exact, see Springer online ref  automorphism group of algebraic variaties. In this example, how do I know $\text{Aut}(A)$ is an algebraic group.
4 Answers
It is not always true that the automorphism group of an algebraic variety has a natural algebraic group structure. For example, the automorphism group of $\mathbb{A}^2$ includes all the maps of the form $(x,y) \mapsto (x, y+f(x))$ where $f$ is any polynomial. I haven't thought through how to say this precisely in terms of functors, but this subgroup morally should be a connected infinite dimensional object, and is thus not a subobject of an algebraic group.
On the other hand, I believe that the automorphism group of a projective algebraic variety, $X$, can be given the structure of algebraic group in a fairly natural way. This is something I've thought about myself, but not written down a careful proof nor found a reference for: For any automorphism $f$ of $X$, consider the graph of $f$ as a subscheme of $X \times X$, and thus a point of the Hilbert scheme of $X\times X$. In this way, we get an embedding of point sets from $\mathrm{Aut}(X)$ into $\mathrm{Hilb}(X\times X)$.
I believe that it should be easy to show that (1) $\mathrm{Aut}(X)$ is open in $\mathrm{Hilb}(X\times X)$, and thus acquires a natural scheme structure and (2) composition of automorphisms is a map of schemes.

2$\begingroup$ More generally, if $X$ and $Y$ are projective, then $Hom(X, Y)$ is an open subscheme of $Hilb(X \times Y)$. If you want a reference, this is carried out in Section 1.1 of Koll\'ar's book, Rational Curves on Algebraic Varieties. $\endgroup$– mdelandDec 15, 2009 at 19:58

$\begingroup$ @ David: It is probably just my ignorance, but why does being an algebraic group imply finite dimensional? For example shouldn't $Spec \; k[b_0, b_1, \dots]$ (infinitely many generators) be an algebraic group where the group law comes from adding, i.e. sending $b_i$ to $x_i + y+i$ in $k[x_1, x_2, \dots, y_1, y_2, \dots]$ ? $\endgroup$ Feb 25, 2010 at 17:11

2$\begingroup$ @Chris: I think what is usually meant as an "algebraic group" is group object in the category of algebraic varieties, as opposed to a "group scheme" which is a group object in the category of (not necessarily locally of finite type) schemes. $\endgroup$– QfwfqOct 26, 2010 at 21:02

9$\begingroup$ It is false that Aut(X) is always an algebraic group, when $X$ is a projective algebraic variety, as it seems guessed in the answer. Take for example two general cubics in the plane and blowup the nine points of intersection, the automorphism group of the smooth projective rational surface obtained is a finite extension of $\mathbb{Z}^8$ and is certainly not an algebraic group. $\endgroup$ Nov 17, 2012 at 17:07
This is really a comment on Pete's comment for Mikhail's answer, but I am making it an answer because it raises a question which I think should be more widely known.
The construction of Autscheme uses the entire Hilbert scheme, which has countably many components (due to varying Hilbert polynomials), and it is not obvious how many of these intervene in the construction. Mikhail's answer illustrates the basic example showing it can be infinite. But this leads to the following problem (suggested by what is known about NeronSeveri groups, whose construction encounters the same issue via construction of Pic schemes in terms of Hilbert schemes, at least in the original Grothendieck construction):
Q. Does the Aut scheme of a proper scheme over an alg. closed field have finitely generated component group? (This is a more basic question than finiteness, which is really Pete's comment to Mikhail's answer.)
Incredibly, even for smooth projective varieties over C this appears to be a wide open problem!! I have mentioned this to several experts in alg. geom. (including Oort), and nobody has an idea. If anyone has an idea or a solution, please let me know right away.

2

1$\begingroup$ Because I assume people with an interest in this topic will see answers to the initial question above, which led to it naturally, and I don't expect that anyone will have an answer. $\endgroup$– BCnrdFeb 25, 2010 at 15:22

2$\begingroup$ +1: This is a very natural question. I wonder why it is not a more famous open problem in the field. (Not just because people don't want to admit that they don't know the answer, I hope?) $\endgroup$ Feb 26, 2010 at 3:32

$\begingroup$ @BCnrd Sorry if I'm being dense but I don't understand the terminology here. Does "finitely generated component group" means the quotient of the groups scheme by the identity component is finitely generated as an abstract group (mean the points of it)? $\endgroup$ Nov 23, 2017 at 8:16

6$\begingroup$ Since this answer is highly upvoted it may be worth mentioning that this open problem was solved in the negative by John Lesieutre: A projective variety with discrete, nonfinitely generated automorphism group. Invent. Math. 212 (2018), no. 1, 189–211. $\endgroup$– BortJan 8, 2019 at 15:17
In [Matsusaka, T. Polarized varieties, fields of moduli and generalized Kummer varieties of polarized abelian varieties. Amer. J. Math. 80 1958 4582.] it is proved that a nonsingular projective variety has a maximal algebraic group of automorphisms (that is, every group which acts on the variety by automorphisms contains a maximal algebraic subgroup)
[Matsumura, Hideyuki; Oort, Frans. Representability of group functors, and automorphisms of algebraic schemes. Invent. Math. 4 1967 125.] shows the automorphism group (of an algebraic proper scheme over a field) is representable by a group scheme.
If A is an elliptic curve then G (in your notation) is finite. Yet it seems that for a square of an elliptic curve you get something infinite and very far from being algebraic. If $A=B\times B$, $B$ is a 'general' elliptic curve, then it seems that $G(A)=GL_2(\Bbb Z)$; this is a 'large' discrete group. Another example is an abelian variety with complex multplication: you get the group of units in some order in some CMfield over $\Bbb Q$, and this is infinite if this field is not quadratic. I think, in these examples it is not difficult to prove that Aut(A) is not an algebraic group.

3$\begingroup$ I think different sources will differ as to whether or not these examples are algebraic groups. The functors are representable by schemes which are locally, but not globally, of finite type. $\endgroup$ Dec 14, 2009 at 17:09

3$\begingroup$ So far as I know, it is fairly standard to require that an algebraic group be an algebraic variety, hence of finite type (hence finitely many connected components). There is no argument that Aut(A) is a (locally algebraic) group scheme, which I think is what is really being asked for here: ideally this more precise language would have been used in the question itself. (On the other hand, one could ask for conditions on a projective variety V so that Aut(V) is of finite type. Anybody know anything about this?) $\endgroup$ Dec 19, 2009 at 11:38

$\begingroup$ @Pete L. Clarck Doesn't the tangent space at the identity (which is finite dimensional in the projective case by finiteness of cohomology) indicate that the automorphism group scheme has an identity component locally of finite type and by translation the whole group? $\endgroup$ Nov 23, 2017 at 8:26
eom.springer.de
is broken. The article can now be found here: Algebraic variety, automorphism of an  Encyclopedia of Mathematics. (In fact, the link in the post points to the reference [2] in the article, which is the following: T. Matsusaka, "Polarized varieties, fields of moduli and generalized Kummer varieties of polarized Abelian varieties" Amer. J. Math., 80 (1958) pp. 45–82. MR0094360, Zbl 0085.15304.) $\endgroup$