I think, because in the category of schemes, all finite limits exist, the commutative group objects with homomorphisms should form an abelian category. Is this true? And do you know anywhere to cite this?

1$\begingroup$ The category of commutative group schemes of finite type over $k$ is abelian, see SGA3, VIA, Thm. 5.4.2; the category of commutative affine group schemes over $k$ is also abelian, see SGA3, VIA, Prop. 5.4.3 and the answer by J. S. Milne below. $\endgroup$– evgenyMar 8, 2017 at 20:52
5 Answers
The category of commutative affine algebraic group schemes over any field is abelian. More generally, the standard isomorphism theorems in abstract group theory hold in the category of affine algebraic group schemes over a field. See, for example, Chapt 1 of my online notes. Over other bases, this need not be true. If you are foolish enough to work with reduced algebraic group schemes (Borel, Humphreys, Springer, et al) then you run into all sorts of problems.
Added: In 1962, Cartier gave a conference talk in which he noted that the standard isomorphism theorems etc. fail with the usual definition of algebraic groups (no nilpotents), but then observed that the "preceding difficulties vanish" when one works with schemes. As far as I know, this is the first statement in print. Of course, it is all covered in the 196364 GrothendieckDemazure seminar (SGA3).
Over arbitrary bases, you can still form kernels, but quotients are a problem because the subgroup need not be flat. As noted, there are also problems when you drop the condition "affine and finite type".
When you don't allow nilpotents (i.e., when you work in the category of reduced group schemes), the Frobenius map $\mathbb{A}^1\to \mathbb{A}^1$ (this is a homomorphism when $\mathbb{A}^1$ is regarded as the additive group scheme) is mono and epi but not an isomorphism (no inverse). In the full category of affine algebraic group schemes, it has a kernel, namely, the finite group scheme $\alpha_p$, so there is no problem.

$\begingroup$ @Milne: Concerning your last sentence: Yes, it has a kernel. But what is its cokernel? $\endgroup$ Sep 11, 2010 at 13:28

$\begingroup$ @Andre Henriques: It is surjective, so the cokernel is trivial. So it identifies the quotient of the additive group by the Frobenius kernel with the additive group. $\endgroup$– t3sujiSep 12, 2010 at 13:23

$\begingroup$ Tank you t3suji. I now understand my mistake, and edited my answer accordingly. $\endgroup$ Sep 12, 2010 at 14:07

3$\begingroup$ It might be worthwile to note that if the field has characteristic $0$, then being foolish does no harm (all group schemes are reduced). $\endgroup$– jmcJan 22, 2015 at 12:38
This is not true.
It fails for essentially the same reason that the category of topological commutative groups fail to be an abelian category. For simplicity let's work over an algebraically closed field k. Consider the additive group over k, whose underlying scheme is the affine line $\mathbb{A}^1 = Spec \; k[x]$. Also consider the scheme X which is a disjoint union of points $Spec \; k$, where the disjoint union is over all points in $\mathbb{A}^1$.
There is a canonical map $X \to \mathbb{A}^1$ and this induces a group structure on X, making it a group scheme as well. The kernel and cokernel of this map are both zero, yet it is clearly not an isomorphism.
Note that X is not an affine scheme, so this does not contradict Milne's answer.
In characteristic zeroand for affine schemes, though, it does work. This is proved in Sweedler's book on Hopf algebras.
This is not true.
The Frobenius homomorphism $(Spec(\mathbb{F}_p[x]),+) \to (Spec(\mathbb{F}_p[x]),+)$ that sends $x$ to $x^p$ has a trivial cokernel. Yet, it is not an isomorphism.
Edit: Ah! I now understand my mistake: it's ok to have nonisomorphisms with trivial kernels.
I must have spent too much time thinking about triangulated categories: in a triangulated category, a map with trivial cofiber is automotically an isomorphism.

$\begingroup$ Here you endow $Spec \mathbb{F}_p[x]$ with the additive group scheme structure? $\endgroup$– QfwfqSep 9, 2010 at 14:02

6$\begingroup$ But doesn't it have a nontrivial kernel, though? $\endgroup$ Sep 9, 2010 at 14:04

$\begingroup$ @Leonid: Correct. The kernel is alpha_p (see JS Milne's answer). $\endgroup$ Sep 12, 2010 at 12:18
should be not true! What is the cokernel of the map $\mathbb{Z}\rightarrow A$ with $A$ an abelian variety and $\mathbb{Z}$ the constant group scheme represented by the ring of integers? But if you consider only commutative algebraic group scheme over a field $k$(i.e. of finite type over $k$), then it should be abelian.

3$\begingroup$ How do your $\mathbb{Z}$ and $A$ live over the same base? I don't understand how $\mathbb{Z}$ is a group object in the same category as in which $A$ is a group object. Please clarify. Moreover, as Milne indicated, your last sentence needs the word “affine” somewhere. $\endgroup$– jmcJan 22, 2015 at 12:33

$\begingroup$ $\mathbb{Z}$ denotes the group scheme over some base represented by the constant additive group $\mathbb{Z}$. Having a morphism is the same as having a point of $A$ over the base. $\endgroup$– HeerJan 24, 2015 at 16:48

$\begingroup$ For the category to be abelian, maybe one also need $k$to be perfect. Isn't the case of alg. closed field in Oort's book "comm. gp. sch."? $\endgroup$– HeerJan 25, 2015 at 9:00