It is not always true that the automorphism group of an algebraic variety has a natural algebraic group structure. For example, the automorphism group of $\mathbb{A}^2$ includes all the maps of the form $(x,y) \mapsto (x, y+f(x))$ where $f$ is any polynomial. I haven't thought through how to say this precisely in terms of functors, but this subgroup morally should be a connected infinite dimensional object, and is thus not a subobject of an algebraic group.
On the other hand, I believe that the automorphism group of a projective algebraic variety, $X$, can be given the structure of algebraic group in a fairly natural way. This is something I've thought about myself, but not written down a careful proof nor found a reference for: For any automorphism $f$ of $X$, consider the graph of $f$ as a subscheme of $X \times X$, and thus a point of the Hilbert scheme of $X$. X\times X$. In this way, we get an embedding of point sets from $\mathrm{Aut}(X)$ into $\mathrm{Hilb}(X)$. \mathrm{Hilb}(X\times X)$.
I believe that it should be easy to show that (1) $\mathrm{Aut}(X)$ is open in $\mathrm{Hilb}(X)$, \mathrm{Hilb}(X\times X)$, and thus acquires a natural scheme structure and (2) composition of automorphisms is a map of schemes.
