It is not always true that the automorphism group of an algebraic variety has a natural algebraic group structure. For example, the automorphism group of $\mathbb{A}^2$ includes all the maps of the form $(x,y) \mapsto (x, y+f(x))$ where $f$ is any polynomial. I haven't thought through how to say this precisely in terms of functors, but this subgroup morally should be a connected infinite dimensional object, and is thus not a subobject of an algebraic group.
On the other hand, I believe that the automorphism group of a projective algebraic variety, $X$, can be given the structure of algebraic group in a fairly natural way. This is something I've thought about myself, but not written down a careful proof nor found a reference for: For any automorphism $f$ of $X$, consider the graph of $f$ as a subscheme of $X \times X$, and thus a point of the Hilbert scheme of $X$. X\times X$. In this way, we get an embedding of point sets from$\mathrm{Aut}(X)$into$\mathrm{Hilb}(X)$. \mathrm{Hilb}(X\times X)$.
I believe that it should be easy to show that (1) $\mathrm{Aut}(X)$ is open in $\mathrm{Hilb}(X)$, \mathrm{Hilb}(X\times X)$, and thus acquires a natural scheme structure and (2) composition of automorphisms is a map of schemes. 1 It is not always true that the automorphism group of an algebraic variety has a natural algebraic group structure. For example, the automorphism group of$\mathbb{A}^2$includes all the maps of the form$(x,y) \mapsto (x, y+f(x))$where$f$is any polynomial. I haven't thought through how to say this precisely in terms of functors, but this subgroup morally should be a connected infinite dimensional object, and is thus not a subobject of an algebraic group. On the other hand, I believe that the automorphism group of a projective algebraic variety,$X$, can be given the structure of algebraic group in a fairly natural way. This is something I've thought about myself, but not written down a careful proof nor found a reference for: For any automorphism$f$of$X$, consider the graph of$f$as a subscheme of$X \times X$, and thus a point of the Hilbert scheme of$X$. In this way, we get an embedding of point sets from$\mathrm{Aut}(X)$into$\mathrm{Hilb}(X)$. I believe that it should be easy to show that (1)$\mathrm{Aut}(X)$is open in$\mathrm{Hilb}(X)\$, and thus acquires a natural scheme structure and (2) composition of automorphisms is a map of schemes.