Here's a sketch of an argument that for $G$ abelian, and for any twisting cocycle, the link invariant depends only on the linking matrix of the link.
Step 1. The irreducible representations of the annular category mentioned above are all 1-dimensional. (This is the category whose morphisms are maps of an annulus into BG.)
Step 2. If $\alpha$ and $\beta$ are two such irreps, then $\alpha\otimes\beta$ is again a 1-dimensional irrep (as opposed to a direct sum of several irreps). Here the tensor product is induced by a pair of pants.
Step 3. It follows that the Hilbert space of a 4-punctured sphere, with punctures labeled by
$\alpha, \beta, \beta^*, \alpha^*$, is 1-dimensional. (cf. Witten's famous argument that in the case of $SU(2)$ Chern-Simons theory the analogous Hilbert space is 2-dimensional.)
Step 4. It follows that there is a skein relation saying that the path integral of a overcrossing is a scalar multiple of the path integral of an undercrossing. (Again cf. Witten in the SU(2) case, where one gets a three term skein relation.) The over- and undercrossings live in a 3-ball whose boundary is the sphere of Step 3.
Step 5. Using the skein relation of Step 4, it follows that the link invariant depends only on the linking matrix and the labels of the components of the link.