I am not sure that all automorphism groups of algebraic varieties have natrual algebraic group structure. But if the automorphism group of a variety has algebraic group structure, how do I know the automorphism group is an algebraic group. For example, the automorphism group of an elliptic curve $A$ is an extension of the group $G$ of automorphisms which preserve the structure of the elliptic curve, by the group $A(k)$ of translations in the points of $A$, i.e. the sequence of groups $0\to A(k)\to \text{Aut}(A)\to G \to 0$ is exact, see Springer online ref  automorphism group of algebraic variaties. In this example, how do I know $\text{Aut}(A)$ is an algebraic group.
It is not always true that the automorphism group of an algebraic variety has a natural algebraic group structure. For example, the automorphism group of $\mathbb{A}^2$ includes all the maps of the form $(x,y) \mapsto (x, y+f(x))$ where $f$ is any polynomial. I haven't thought through how to say this precisely in terms of functors, but this subgroup morally should be a connected infinite dimensional object, and is thus not a subobject of an algebraic group. On the other hand, I believe that the automorphism group of a projective algebraic variety, $X$, can be given the structure of algebraic group in a fairly natural way. This is something I've thought about myself, but not written down a careful proof nor found a reference for: For any automorphism $f$ of $X$, consider the graph of $f$ as a subscheme of $X \times X$, and thus a point of the Hilbert scheme of $X\times X$. In this way, we get an embedding of point sets from $\mathrm{Aut}(X)$ into $\mathrm{Hilb}(X\times X)$. I believe that it should be easy to show that (1) $\mathrm{Aut}(X)$ is open in $\mathrm{Hilb}(X\times X)$, and thus acquires a natural scheme structure and (2) composition of automorphisms is a map of schemes. 


This is really a comment on Pete's comment for Mikhail's answer, but I am making it an answer because it raises a question which I think should be more widely known. The construction of Autscheme uses the entire Hilbert scheme, which has countably many components (due to varying Hilbert polynomials), and it is not obvious how many of these intervene in the construction. Mikhail's answer illustrates the basic example showing it can be infinite. But this leads to the following problem (suggested by what is known about NeronSeveri groups, whose construction encounters the same issue via construction of Pic schemes in terms of Hilbert schemes, at least in the original Grothendieck construction): Q. Does the Aut scheme of a proper scheme over an alg. closed field have finitely generated component group? (This is a more basic question than finiteness, which is really Pete's comment to Mikhail's answer.) Incredibly, even for smooth projective varieties over C this appears to be a wide open problem!! I have mentioned this to several experts in alg. geom. (including Oort), and nobody has an idea. If anyone has an idea or a solution, please let me know right away. 


In [Matsusaka, T. Polarized varieties, fields of moduli and generalized Kummer varieties of polarized abelian varieties. Amer. J. Math. 80 1958 4582.] it is proved that a nonsingular projective variety has a maximal algebraic group of automorphisms (that is, every group which acts on the variety by automorphisms contains a maximal algebraic subgroup) [Matsumura, Hideyuki; Oort, Frans. Representability of group functors, and automorphisms of algebraic schemes. Invent. Math. 4 1967 125.] shows the automorphism group (of an algebraic proper scheme over a field) is representable by a group scheme. 


If A is an elliptic curve then G (in your notation) is finite. Yet it seems that for a square of an elliptic curve you get something infinite and very far from being algebraic. If $A=B\times B$, $B$ is a 'general' elliptic curve, then it seems that $G(A)=GL_2(\Bbb Z)$; this is a 'large' discrete group. Another example is an abelian variety with complex multplication: you get the group of units in some order in some CMfield over $\Bbb Q$, and this is infinite if this field is not quadratic. I think, in these examples it is not difficult to prove that Aut(A) is not an algebraic group. 

