Here is a sketch proof.
Step 1: For sensible spaces or spectra (connected, finite type) $X$, $H^*(X;\tau) $ will have exponent $p$ for all the coefficient groups $\tau$ you list exactly when the image of the Bockstein $\beta: H^*(X;\mathbb Z/p) \rightarrow H^{*+1}(X;\mathbb Z/p)$ equals the kernel of $\beta$. This can be proved by fooling around with the various coefficient sequences defining the Bockstein. Bill Browder developed this idea into the `Bockstein spectral sequence', which he made good use of.
Step 2: So now we need to look at $H^*(H\pi;\mathbb Z/p)$, viewed graded vector space with the Bockstein action, and we want to show that this module is `$\beta$--acylic'. The Kunneth theorem applies, and we can reduce to the cases when $\pi = \mathbb Z$ or $\pi=\mathbb Z/p^m$.
Now we use the calculations of Serre, et. al. First of all, $H^*(H\mathbb Z/p;\mathbb Z/p) = A$, the mod $p$ Steenrod algebra. When written with Serre's admissible basis, one can see that $A$ is $\beta$--acyclic. (In fact, $A$ is forced by general Hopf algebra theory to be free as a module over the exterior algebra on $\beta$.) Then $H^*(H\mathbb Z;\mathbb Z/p) = A/A\beta$, which one checks is also $\beta$--acyclic ($\beta$ acts on the left). Finally, if $m>1$, then $H^*(H\mathbb Z/p^m;\mathbb Z/p) = A/A\beta \oplus \Sigma A/A\beta$, and so is also $\beta$--acyclic.
So this is what Rudyak meant.
The canonical mod 2 reference is Serre, Jean-Pierre Groupes d'homotopie et classes de groupes abéliens. (French) Ann. of Math. (2) 58, (1953). 258–294. Odd primes would have been later, but not by much.
A little bit more added later $\dots$
The cofibration sequence $H\mathbb Z \rightarrow H\mathbb Z \rightarrow H\mathbb Z/p$ induces a short exact sequence $0 \rightarrow \Sigma H^*(H\mathbb Z; \mathbb Z/p) \rightarrow A \rightarrow H^*(H\mathbb Z; \mathbb Z/p) \rightarrow 0$. It is not hard to see that the short exact sequence $0 \rightarrow im \beta \rightarrow A \rightarrow coker \beta \rightarrow 0$ maps to this (with the identity in the middle slot). The fact that $A$ is $\beta$--acyclic ($im \beta = ker \beta$) implies that $im \beta$ and $coker \beta$ are the `same size', as graded vector spaces (up to a suspension). This forces the map between the short exact sequences to be an isomorphism, and so one has $H^*(H\mathbb Z; \mathbb Z/p) = coker \beta = A/A\beta$.