I'm not sure why this hasn't been mentioned yet, but the category of abelian group objects in smooth proper geometrically integral schemes is not abelian, already if $S = \operatorname{Spec} k$. Indeed, if $A$ is any abelian variety, then the map $[n] \colon A \to A$ for $n > 1$ is a monomorphism and an epimorphism:
- As the OP explained here, it is an epimorphism of schemes since it is surjective on points and injective on sections (being a dominant morphism of integral schemes). Thus it is also an epimorphism of group schemes.
- If $f \colon G \to A$ is a map such that $G \to A \stackrel{[n]}\to A$ is trivial, then $f(G)$ lands in $A[n]$ scheme-theoretically. But if $G$ is geometrically integral, this implies $f(G)$ lands in the trivial subgroup $\boldsymbol 1$, again scheme-theoretically.
Thus, $[n] \colon A \to A$ is both a monomorphism and an epimorphism, but it clearly does not have an inverse (even as schemes) when $n > 1$.
Dropping "geometrically integral" doesn't help: if $\operatorname{char} k = p > 0$ and $A$ is supersingular, the same example with $n = p$ shows that looking at smooth abelian group schemes is not enough. Dropping both does work if you further impose finite type hypotheses: the category of abelian group schemes of finite type over a field $k$ is abelian; see this post and the references therein. Then the proper ones form a Serre subcategory, hence an abelian category.
It would be interesting to see what versions are true over a general base. It seems hard to me to make quotients if there are no flatness assumptions, but maybe kernels or cokernels of flat group schemes don't want to be flat. I should probably try to read some SGA3...