For a number field F, let E/F be an elliptic curve with CM by a quadratic field K. Let $\rho_\ell: \text{Gal}_F \to \text{Aut}(T_{\ell}E)$ be the $\ell$-adic representation associated to E for some prime $\ell$, and let $\mathcal{G}_{\ell}$ denote its image. According to Theorem 5 in Serre's "Groupes de Lie $\ell$-adiques attaches aux courbes elliptiques" (1966), $\mathcal{G}_{\ell}$ is abelian if and only if K $\subset$ F.
Is it always the case that $\mathcal{G}_{\ell}$ being abelian forces K $\subset$ F, or are there some restrictions on $\ell$ that I haven't been able to find in the paper?