Let $p\geq 3$ be a prime number. The modular curve $X(p)$ can be considered as a connected smooth projective curve over complex numbers. There is a subgroup $\mathrm{PSL}_2(\mathbb{F}_p)$ inside its group of automorphisms (Serre has proved that for $p\geq 7$ it is the full group of automorphisms).

For which $p$ does there exist a geometrically connected smooth projective curve $X$ over $\mathbb{Q}$ such that $X(p)$ is the base change of $X$ and such that all $\mathrm{PSL}_2(\mathbb{F}_p)$-automorphisms of $X(p)$ are base changed from automorphisms of $X$?