Gerry Myerson already answered in the comments, but let me be a little more explicit. Take any number $m\in\mathbb{N}$ with at least two prime factors (for example $6$). Then construct a Cantor set $C=C_m$ similar to the ternary Cantor set, except that instead of keeping two intervals of relative length $1/3$ we keep two intervals of relative length $1/m$ at each step. Alternatively, $C_m$ is the set of points in $[0,1]$ whose base $m$ expansion has only digits $0, m-1$. Clearly, no point in $C$ is remotely normal to base $m$.
On the other hand, $C$ carries a natural measure $\mu$, which can be defined by the property that is assigns the same mass $2^{-k}$ to all $2^k$ intervals in the $k$-th stage of the construction ($\mu$ is also a multiple of Hausdorff measure of the appropriate dimension on $C$).
It is then known that $\mu$-almost all points are normal to any base $p$ such that $\log p/\log m$ is irrational. In particular, this is true for all prime numbers $p$. So $\mu$-almost all points are counterexamples to the question.
The proof of this essentially goes back to Cassels [Cassels, J. W. S. On a problem of Steinhaus about normal numbers. Colloq. Math. 7 1959 95--101], who proved the special case $m=3$, by looking at the Fourier transform of $\mu$ along sequences $p^n \xi$. I am quite sure the proof works for any $m$. See also our paper [Hochman, Michael; Shmerkin, Pablo. Equidistribution from fractal measures. Invent. Math. 202 (2015), no. 1, 427--479.] for more discussion and generalizations.