Can Hom_gp(G,H) fail to be representable for affine algebraic groups? Let $G$ and $H$ be affine algebraic groups over a scheme $S$ of characteristic 0 and let $\textbf{Hom}_{S,gp}(G,H)$ be the functor $T \mapsto \text{Hom}\_{T,gp}(G,H)$

Theorem (SGA 3, expose XXIV, 7.3.1(a)): Suppose G is reductive. Then 
  $\textbf{Hom}_{S,gp}(G,H)$ is representable by a scheme.

Can this fail if $G$ is not reductive? I worked out a few example with $G = \mathbb{G}_a$, but they were representable.
 A: $\operatorname{Hom}(\mathbb{G}_a, \mathbb{G}_m)$ is not representable.
Let $R$ be a $\mathbb{Q}$-algebra. I claim that $\operatorname{Hom}(\mathbb{G}_a, \mathbb{G}_m)(\operatorname{Spec} R)$ is {Nilpotent elements of $R$}. Intuitively, all homs are of the form $x\mapsto e^{nx}$ with $n$ nilpotent.
More precisely, the schemes underlying $\mathbb{G}_a$ and $\mathbb{G}_m$ are
$\operatorname{Spec} R[x]$ and $\operatorname{Spec} R[y, y^{-1}]$ respectively. Any hom of schemes is of the form $y \mapsto \sum f_i x^i $for some $f_i$ in $R$. The condition that this be a hom of groups says that
$\sum f_k (x_1+x_2)^k = (\sum f_i x_1^i)(\sum f_j x_2^j)$. Expanding this, $f_{i+j}/(i+j)! = f_i/i! f_j/j!$. So every hom is of the form $f_i = n^i/i!$, and n must be nilpotent so that the sum will be finite.
Now, let's see that this isn't representable. For any positive integer $k$, let $R_k = C[t]/t^k$. The map $x \mapsto e^{tx}$ is in $\operatorname{Hom}(\mathbb{G}_a, \mathbb{G}_m)(\operatorname{Spec} R_k)$ for every $k$. However, if $R$ is the inverse limit of the $R_k$, there is no corresponding map in $\operatorname{Hom}(\mathbb{G}_a, \mathbb{G}_m)(\operatorname{Spec} R)$. So the functor is not representable.
A: There is a reasonable salvage, at least if the base scheme is a field: Hom(G,H) is a direct limit of representable subfunctors.  See Lemma A.8.13 in the book "pseudo-reductive groups" (where it is used to prove that the scheme-theoretic fixed locus for a linearly reductive group acting on a connected reductive group is always reductive (possibly disconnected) provided the base is a field.  
