I think, because in the category of schemes, all finite limits exist, the commutative group objects with homomorphisms should form an abelian category. Is this true? And do you know anywhere to cite this?
The category of commutative affine algebraic group schemes over any field is abelian. More generally, the standard isomorphism theorems in abstract group theory hold in the category of affine algebraic group schemes over a field. See, for example, Chapt 1 of my online notes. Over other bases, this need not be true. If you are foolish enough to work with reduced algebraic group schemes (Borel, Humphreys, Springer, et al) then you run into all sorts of problems. Added: In 1962, Cartier gave a conference talk in which he noted that the standard isomorphism theorems etc. fail with the usual definition of algebraic groups (no nilpotents), but then observed that the "preceding difficulties vanish" when one works with schemes. As far as I know, this is the first statement in print. Of course, it is all covered in the 196364 GrothendieckDemazure seminar (SGA3). Over arbitrary bases, you can still form kernels, but quotients are a problem because the subgroup need not be flat. As noted, there are also problems when you drop the condition "affine and finite type". When you don't allow nilpotents (i.e., when you work in the category of reduced group schemes), the Frobenius map $\mathbb{A}^1\to \mathbb{A}^1$ (this is a homomorphism when $\mathbb{A}^1$ is regarded as the additive group scheme) is mono and epi but not an isomorphism (no inverse). In the full category of affine algebraic group schemes, it has a kernel, namely, the finite group scheme $\alpha_p$, so there is no problem. 


This is not true. It fails for essentially the same reason that the category of topological commutative groups fail to be an abelian category. For simplicity let's work over an algebraically closed field k. Consider the additive group over k, whose underlying scheme is the affine line $\mathbb{A}^1 = Spec \; k[x]$. Also consider the scheme X which is a disjoint union of points $Spec \; k$, where the disjoint union is over all points in $\mathbb{A}^1$. There is a canonical map $X \to \mathbb{A}^1$ and this induces a group structure on X, making it a group scheme as well. The kernel and cokernel of this map are both zero, yet it is clearly not an isomorphism. Note that X is not an affine scheme, so this does not contradict Milne's answer. 


In characteristic zeroand for affine schemes, though, it does work. This is proved in Sweedler's book on Hopf algebras. 


The Frobenius homomorphism $(Spec(\mathbb{F}_p[x]),+) \to (Spec(\mathbb{F}_p[x]),+)$ that sends $x$ to $x^p$ has a trivial cokernel. Yet, it is not an isomorphism. Edit: Ah! I now understand my mistake: it's ok to have nonisomorphisms with trivial kernels. I must have spent too much time thinking about triangulated categories: in a triangulated category, a map with trivial cofiber is automotically an isomorphism. 

