Your belief is correct. A $\mathbb{Z}$-point has to reduce to an $\mathbb{F}_p$-point for all $p$, which kills examples with gcd > 1.
If you want to make this precise, try writing down an explicit description of X by patching affine pieces. All the essential ideas are already there in $\mathbb{A}^1 \backslash 0$: this is the spectrum of $\mathbb{Z}[X, Y] / (XY - 1)$.