Is there an example that the category of sheaves of abelian groups on a site is not an abelian category?
The category of sheaves of abelian groups on a site is always abelian. See Theorem 2.1.4 on page 20 of
which is available online: http://www.math.ubc.ca/~gor/Artin-GT.pdf
Once you construct the sheafification functor in the context of an arbitrary site most of the basic results in the classical theory of sheaves can be carried over to the context of sites without (much) modification.