3 edited body

A simple-minded argument:

Pick any element $g\in G$ of order $p$, the characteristic of $k$, and let $C(g)$ be the centralizer of $g$.

Now $G\setminus C(g)$ is the disjoint union of orbits under the action of the inner automorphism $\iota_g:h\in G\mapsto ghg^{-1}\in G$, and those orbits are of size $p$. The sum of the terms in your sum corresponding to the elements of $g$ in one of those orbits is then zero, for those terms are all equal.

If $h\in C(g)$, then the terms in your sum corresponding to the elements $h$, $gh$, $g^2h$, $\dots$, $g^{p-1}h$ are all equal---because $g$ and $h$ commute, you can take them simutaneously to Jordan canonical form and $g$ has only $1$ as an eigenvalue---so that their sum is also zero.

We have thus partitioned $G$ into, on one hand, the orbits of $\iota_g$ in $G\setminus C(g)$, and, on the other, the cosets of $\langle g\rangle$ in $C(g)$, and checked that the sum of the terms in each part of this partition is zero. Therefore your sum is zero.

NB: Notice that the specific form of the terms in your sum do does not really matter, as long as they it only depend on the eigenvalues of the $g\in G$. Thus, for example, exactly the same reasoning shows that the "other" Molien formula $$\sum_{g\in G}\det(I-t\rho(g))$$ also vanishes.

2 added 285 characters in body

A simple-minded argument:

Pick any element $g\in G$ of order $p$, the characteristic of $k$, and let $C(g)$ be the centralizer of $g$.

Now $G\setminus C(g)$ is the disjoint union of orbits under the action of the inner automorphism $\iota_g:h\in G\mapsto ghg^{-1}\in G$, and those orbits are of size $p$. The sum of the terms in your sum corresponding to the elements of $g$ in one of those orbits is then zero, for those terms are all equal.

If $h\in C(g)$, then the terms in your sum corresponding to the elements $h$, $gh$, $g^2h$, $\dots$, $g^{p-1}h$ are all equal---because $g$ and $h$ commute, you can take them simutaneously to Jordan canonical form and $g$ has only $1$ as an eigenvalue---so that their sum is also zero.

We have thus partitioned $G$ into, on one hand, the orbits of $\iota_g$ in $G\setminus C(g)$, and, on the other, the cosets of $\langle g\rangle$ in $C(g)$, and checked that the sum of the terms in each part of this partition is zero. Therefore your sum is zero.

NB: Notice that the specific form of the terms in your sum do not really matter, as long as they only depend on the eigenvalues of the $g\in G$. Thus, for example, exactly the same reasoning shows that the "other" Molien formula $$\sum_{g\in G}\det(I-t\rho(g))$$ also vanishes.

1

A simple-minded argument:

Pick any element $g\in G$ of order $p$, the characteristic of $k$, and let $C(g)$ be the centralizer of $g$.

Now $G\setminus C(g)$ is the disjoint union of orbits under the action of the inner automorphism $\iota_g:h\in G\mapsto ghg^{-1}\in G$, and those orbits are of size $p$. The sum of the terms in your sum corresponding to the elements of $g$ in one of those orbits is then zero, for those terms are all equal.

If $h\in C(g)$, then the terms in your sum corresponding to the elements $h$, $gh$, $g^2h$, $\dots$, $g^{p-1}h$ are all equal---because $g$ and $h$ commute, you can take them simutaneously to Jordan canonical form and $g$ has only $1$ as an eigenvalue---so that their sum is also zero.

We have thus partitioned $G$ into, on one hand, the orbits of $\iota_g$ in $G\setminus C(g)$, and, on the other, the cosets of $\langle g\rangle$ in $C(g)$, and checked that the sum of the terms in each part of this partition is zero. Therefore your sum is zero.