I'm not sure why this hasn't been mentioned yet, but the category of abelian group objects in smooth proper geometrically integral schemes is not abelian, already if $S = \operatorname{Spec} k$. For example, the map $(-)^n \colon \mathbf G_m \to \mathbf G_m$ for $n > 1$ is a monomorphism and an epimorphism:

 - As the OP explained [here](https://mathoverflow.net/q/56564/82179), it is an epimorphism of schemes since it is surjective on points and injective on sections. Thus it is also an epimorphism of group schemes.
 - If $f \colon G \to \mathbf G_m$ is a map such that $G \to \mathbf G_m \stackrel{(-)^n}\to \mathbf G_m$ is trivial, then $f(G)$ lands in $\boldsymbol\mu_n$ scheme-theoretically. But if $G$ is geometrically integral, this implies $f(G)$ lands in the trivial subgroup $\boldsymbol 1$, again scheme-theoretically.

Thus, $(-)^n \colon \mathbf G_m \to \mathbf G_m$ is both a monomorphism and an epimorphism, but it clearly does not have an inverse (even as schemes) when $n > 1$.

Dropping "geometrically integral" doesn't help: the same example with $n = p$ for $\operatorname{char} k = p > 0$ shows that looking at *smooth* abelian group schemes is not enough. Dropping both does work if you further impose finite type hypotheses: the category of abelian group schemes of finite type over a field $k$ is abelian; see [this post](https://mathoverflow.net/q/38168/82179) and the references therein.

It would be interesting to see what versions are true over a general base. It seems hard to me to make quotients if there are no flatness assumptions, but maybe kernels or cokernels of flat group schemes don't want to be flat. I should probably try to read some SGA3...