For the primes in the denominator, there is an amusing heuristic based on the fact that $n \equiv n^{-1}\pmod p$ holds for all $n\geq 1$ (coprime to $p$) only for $p=2$ and $p=3$. So for these primes the series is like the series $1+1/2+1/3+\cdots$ for $\zeta(1)$, which has a true pole...