Background: Let $G$ be a finite group, and $\mathrm{Vect}_G$ be the category of finite dimensional $G$-graded vector spaces over some algebraically closed field $k$ of char 0. It is well-known that $\mathrm{Vect}_G$ is a spherical fusion category with a representative set of simple objects given by $I=\{I_x\}_{x\in G}$, where $(I_x)_g=\delta_{xg}k$. The Drinfeld center $Z(\mathrm{Vect}_G)$ of this category is therefore modular, according to a fundamental theorem of Michael Müger, which states that the Drinfeld center of a spherical fusion category is modular. As a result, the S-matrix of $Z(\mathrm{Vect}_G)$ should be invertible.
Problem: To make things easier, we further assume $G$ to be abelian. Without using the theorem of Müger, how to prove the invertibility of the S-matrix of $Z(\mathrm{Vect}_G)$?
Here is my approach.
Approach: Let $\mathcal{C}=\mathrm{GrRep}(G)$ be the category of finite dimensional $G$-graded representations of $G$. A typical element of $\mathrm{GrRep}(G)$ is a $G$-graded vector space $V\in\mathrm{Vect}_G $ together with a group homomorphism $\rho:G\to \mathrm{Hom}_{\mathrm{Vect}_G}(V,V)$. It is easy to see that $\mathcal{C}$ is fusion with a finite representative set of simple objects $$J=\{(I_x,\rho_\phi):x\in G, \phi\in \mathrm{Hom}_{\mathrm{Grp}}(G,k^\times),\rho_\phi(g)=\phi(g)\mathrm{id}_{I_x} \}. $$ Next, I proved that $Z(\mathrm{Vect}_G)\cong \mathcal{C}$. It remains to show that the S-matrix of $\mathcal{C}$ w.r.t. the basis $J$ is invertible. Note that this is a symmetric matrix of size $|G|^2$. It turns out that even if in the simplest case that $G$ is a cyclic group of order $p$, this matrix is not easy to handle for me.
Did I take a wrong way? Or there are some tricks in linear algebra I missed. Any comments or helps would be very much appreciated.