My question got answered in the comments, so I thought best to write a small answer for it here:
If $C$ is a rigid monoidal category, then the forgetful functor $U: Z(C) \rightarrow C $ creates colimits and limits: any (co)limit of the underlying objects of a diagram in $Z(C)$ will have an induced braiding satisfying the nessecary conditions. This follows from $A\otimes - $ and $-\otimes A$ preserving (co)limites.
Hence, if $C$ is abelian, then the center is finitely complete and cocomplete and naturally automatically additive. The only thing remaining is to check if any mono is a kernel and any epi is a cokernel which I was stuck at, which just follows from the fact that in a finitely (co)complete category, monos (epis) are just pullbacks and pushforwards, so they are preserved by $U$. If $m:(A,\sigma)\rightarrow (B,\tau) $ is a monomorphism in the center, then $m$ is a mono in $C$ and is the kernel of $f:B\rightarrow coker(f)$ and as mentioned above $coker(f)$ has an induced braiding compatible with $\sigma $ and $\tau $ and $f$.