When do the Galois reps of modular forms have open image?

Suppose f is a newform (with coefficients generating some number field E), and $\rho_{f,\lambda}: {\rm Gal}(\overline{\mathbb{Q}} / \mathbb{Q}) \to {\rm GL}_2(E_\lambda)$ the associated Galois rep (for some prime $\lambda$ of E). When does $\rho$ have open image in ${\rm GL}_2(E_\lambda)$?

This clearly isn't the case if f has weight 1, or if f is of CM type; and I gather that it's a theorem of Serre that if f is attached to an elliptic curve, then not having CM is sufficient. What's known about this question in general?

-
This is undoubtedly a difficult question in general. If $M$ is the motive associated to $f$, it determines a Mumford-Tate group $MT\subset GL(M_B)$ ($M_B$ is the Betti realization), and the $\mathbb{Q}_l$ representation of the Galois group is conjectured to have open image inside $MT(\mathbb{Q}_l)$. Experts in Hodge theory (not me) should be able to compute $MT$, but very little is known about the conjecture. Serre's theorem is a very special case. I understand more developments (for abelian varieties) are in his College de France lectures in 1985. See also papers of Richard Pink. –  Minhyong Kim May 10 '10 at 10:37
The version of Ribet's result that I recall (not so well; it has been a while since I thought about these things) is that for any modular abelian variety, the $\ell$-adic image of Galois is open in $MT(\mathbb{Q}_{\ell})$, and the latter group is always determined only by the endomorphism algebra of the abelian variety, i.e., the only restriction is that (after finite base change) the image of Galois commutes with $\operatorname{End}(A) \otimes \mathbb{Q}_{\ell}$. Does this sound right to the cognoscenti? –  Pete L. Clark May 10 '10 at 15:56
I haven't looked through Ribet's article in detail (at least, not recently), but it seems to me the OP's statement cannot be correct in all cases. Consider the case of a newform with real quadratic Fourier field $E$ such that the corresponding abelian variety has quaternionic multiplication (such things exist!). Then the image of any $\ell$-adic Galois representation is at most $4$-dimensional (and it is a theorem of Ohta that it is exactly $4$-dimensional). When $\lambda$ is inert in $E$, this is smaller than is claimed above. –  Pete L. Clark May 10 '10 at 15:49