You have noted (accurately) that $G(p) =G$ unless $G$ has a cyclic Sylow $p$-subgroup. However, it is also clear that when $G$ has a cyclic Sylow $p$-subgroup and $G(p) \neq G$
the group $G$ has a normal $p$-complement. For, otherwise, we have by Burnside's transfer theorem, there is a $p$-regular element $x \in N_{G}(P) \backslash C_{G}(P)$. Now we have $P = [P,x] \times C_{P}(x).$ Since $P$ is cyclic and $P \neq C_{P}(x),$ we must have $P = [P,x]$ and $C_{P}(x) = 1.$ But then $P \leq G^{\prime},$ a contradiction, since $[G:G(p)] =p$ and $G(p)$ is clearly a normal subgroup.

On the other hand, if $G$ has a normal $p$-complement and cyclic non-trivial Sylow $p$-subgroup, the clearly $G$ has a normal subgroup of index $p$, and that that subgroup is indeed $G(p)$.

The conclusion (when $|G|$ has order divisible by $p,$ is that $G(p)$ is a subgroup of $G$ if and only if one of the following cases occur:

$G(p) = G$, or, $G$ has a normal $p$-complement and a cyclic Sylow $p$-subgroup.