Let $f=\sum_{n=1}^\infty a_nq^n$ be a newform of level $N$ and weight $k\ge 2$. Suppose that $f$ is a CM modular form in the sense of Ribet: there exists a quadratic character $\varphi$ such that $$a_p = \varphi(p)a_p$$for all primes $p$ in a set of primes of density $1$. Let $K$ be the quadratic field cut out by $\varphi$.

Is there a down-to-earth explanation of why $K$ must be an imaginary quadratic field?