I think this is very false. A counterexample is given by the spaces $\overline M_{g,n}$, which are certainly smooth and proper but far from rational in the large $g$ limit (or, for $g > 0$, in the large $n$ limit). The original references here are, I guess, Deligne (for $\overline M_{1,11}$) and Harris-Mumford (for $\overline M_{25}$). If you are troubled by the fact that this is a stack and not a variety, then you are still fine since $\overline M_{g,n}$ is a finite cover of a smooth proper scheme.
Kevin Buzzard's hint with the Ramanujan $\Delta$ function is relevant here; indeed, $H^{11,0}(\overline M_{1,11})$ is nonzero, and the $\ell$-adic Galois representation corresponding to $H^{11,0}\oplus H^{0,11}$ is the representation attached to $\Delta$.
Edit: My answer to the question Is the moduli space of curves defined over the field with one element? contains some more detailed information about these things.