It is easier to describe non-good numbers with respect to a given prime $p$. For each such number $k$, there exists a prime $q$ such that $q\mid k$ and $q\mid (p^k - 1)$. It follows that $k$ is multiple of $m_p(q):=q\cdot \mathrm{ord}_q(p)$, where $\mathrm{ord}_q(p)$ is the multiplicative order of $p$ modulo $q$ (which is a divisor of $q-1$). Hence, the set of non-good $k$ is formed by the union $$\bigcup_{q\in\mathbb{P}\atop q\ne p} m_p(q)\mathbb{Z}.$$
It also follows that $k$ non-good w.r.t. to any prime $p\geq3$ form the set $$\bigcup_{q\in\mathbb{P}} q(q-1)\mathbb{Z}.$$