I think, because in the category of schemes, all finite limits exist, the commutative group objects with homomorphisms should form an abelian category. Is this true? And do you know anywhere to cite this?

1$\begingroup$ The category of commutative group schemes of finite type over $k$ is abelian, see SGA3, VIA, Thm. 5.4.2; the category of commutative affine group schemes over $k$ is also abelian, see SGA3, VIA, Prop. 5.4.3 and the answer by J. S. Milne below. $\endgroup$ – evgeny Mar 8 '17 at 20:52
The category of commutative affine algebraic group schemes over any field is abelian. More generally, the standard isomorphism theorems in abstract group theory hold in the category of affine algebraic group schemes over a field. See, for example, Chapt 1 of my online notes. Over other bases, this need not be true. If you are foolish enough to work with reduced algebraic group schemes (Borel, Humphreys, Springer, et al) then you run into all sorts of problems.
Added: In 1962, Cartier gave a conference talk in which he noted that the standard isomorphism theorems etc. fail with the usual definition of algebraic groups (no nilpotents), but then observed that the "preceding difficulties vanish" when one works with schemes. As far as I know, this is the first statement in print. Of course, it is all covered in the 196364 GrothendieckDemazure seminar (SGA3).
Over arbitrary bases, you can still form kernels, but quotients are a problem because the subgroup need not be flat. As noted, there are also problems when you drop the condition "affine and finite type".
When you don't allow nilpotents (i.e., when you work in the category of reduced group schemes), the Frobenius map $\mathbb{A}^1\to \mathbb{A}^1$ (this is a homomorphism when $\mathbb{A}^1$ is regarded as the additive group scheme) is mono and epi but not an isomorphism (no inverse). In the full category of affine algebraic group schemes, it has a kernel, namely, the finite group scheme $\alpha_p$, so there is no problem.

$\begingroup$ @Milne: Concerning your last sentence: Yes, it has a kernel. But what is its cokernel? $\endgroup$ – André Henriques Sep 11 '10 at 13:28

$\begingroup$ @Andre Henriques: It is surjective, so the cokernel is trivial. So it identifies the quotient of the additive group by the Frobenius kernel with the additive group. $\endgroup$ – t3suji Sep 12 '10 at 13:23

$\begingroup$ Tank you t3suji. I now understand my mistake, and edited my answer accordingly. $\endgroup$ – André Henriques Sep 12 '10 at 14:07

3$\begingroup$ It might be worthwile to note that if the field has characteristic $0$, then being foolish does no harm (all group schemes are reduced). $\endgroup$ – jmc Jan 22 '15 at 12:38
This is not true.
It fails for essentially the same reason that the category of topological commutative groups fail to be an abelian category. For simplicity let's work over an algebraically closed field k. Consider the additive group over k, whose underlying scheme is the affine line $\mathbb{A}^1 = Spec \; k[x]$. Also consider the scheme X which is a disjoint union of points $Spec \; k$, where the disjoint union is over all points in $\mathbb{A}^1$.
There is a canonical map $X \to \mathbb{A}^1$ and this induces a group structure on X, making it a group scheme as well. The kernel and cokernel of this map are both zero, yet it is clearly not an isomorphism.
Note that X is not an affine scheme, so this does not contradict Milne's answer.
This is not true.
The Frobenius homomorphism $(Spec(\mathbb{F}_p[x]),+) \to (Spec(\mathbb{F}_p[x]),+)$ that sends $x$ to $x^p$ has a trivial cokernel. Yet, it is not an isomorphism.
Edit: Ah! I now understand my mistake: it's ok to have nonisomorphisms with trivial kernels.
I must have spent too much time thinking about triangulated categories: in a triangulated category, a map with trivial cofiber is automotically an isomorphism.

$\begingroup$ Here you endow $Spec \mathbb{F}_p[x]$ with the additive group scheme structure? $\endgroup$ – Qfwfq Sep 9 '10 at 14:02

6$\begingroup$ But doesn't it have a nontrivial kernel, though? $\endgroup$ – Leonid Positselski Sep 9 '10 at 14:04

$\begingroup$ @Leonid: Correct. The kernel is alpha_p (see JS Milne's answer). $\endgroup$ – André Henriques Sep 12 '10 at 12:18
In characteristic zeroand for affine schemes, though, it does work. This is proved in Sweedler's book on Hopf algebras.
should be not true! What is the cokernel of the map $\mathbb{Z}\rightarrow A$ with $A$ an abelian variety and $\mathbb{Z}$ the constant group scheme represented by the ring of integers? But if you consider only commutative algebraic group scheme over a field $k$(i.e. of finite type over $k$), then it should be abelian.

3$\begingroup$ How do your $\mathbb{Z}$ and $A$ live over the same base? I don't understand how $\mathbb{Z}$ is a group object in the same category as in which $A$ is a group object. Please clarify. Moreover, as Milne indicated, your last sentence needs the word “affine” somewhere. $\endgroup$ – jmc Jan 22 '15 at 12:33

$\begingroup$ $\mathbb{Z}$ denotes the group scheme over some base represented by the constant additive group $\mathbb{Z}$. Having a morphism is the same as having a point of $A$ over the base. $\endgroup$ – Heer Jan 24 '15 at 16:48

$\begingroup$ For the category to be abelian, maybe one also need $k$to be perfect. Isn't the case of alg. closed field in Oort's book "comm. gp. sch."? $\endgroup$ – Heer Jan 25 '15 at 9:00