Let $X\to\mathrm{Spec}\:\mathbb{F}_p$ be a smooth proper morphism. Is there a closed immersion $X\to Y$ where $Y$ is flat of finite type over $\mathbb{Z}/p^2\mathbb{Z}$?
As mentioned in the comments if $X$ is projective the problem is solved immediately. So a counterexample would have $\mathrm{dim}(X)\geq 3$.