If there were a newform which didn't have potentially good reduction and whose associated abelian variety were isogenous to a subvariety of $Jac(X(p))$ then the associated local automorphic representation would have to be a twist of the Steinberg. The character that one is twisting by would have to have conductor 1 or $p$ (otherwise the conductor of the local automorphic rep at $p$ would be bigger than $p^2$) and there's a global character which looks like this locally on inertia; untwisting by this global char shows that the twisted modular form contributes to the cohomology of $X_0(p)$. So if $X_0(p)$ has genus 0 then $Jac(X(p))$ has potentially good reduction at $p$. I don't think this quite proves that $X(p)$ has good reduction at $p$ but it answers Francois' question at least. The implicit claim in the question that $X_0(p)$ has good reduction at $p$ because it has genus 0 isn't quite a complete argument: twists of the projective line can have bad reduction at some primes, but Will is of course OK because $X_0(p)$ has cusps and hence points.