Question

Suppose one has an elliptic curve $E$ over $\mathbb{Q}$ with conductor $N < k^3$ for some (large) positive $k$, with the property that its Fourier coefficients satisfy $$ a_p=0, \; \mbox{ for all } \; p \equiv -1 \mod{4}, \; \; \; k/3 < p < k. $$ Can one prove unconditionally that $E$ has CM? This follows from work of Serre (under GRH) or Elkies (under something like Szpiro's conjecture), since otherwise we'd have a surplus of supersingular primes. It does not appear to be a consequence of, say, Serre's argument without additional hypotheses (though I'd be happy to be wrong on this score).