Let us assume that $E$ has complex multiplication by a maximal order $\mathcal{O}$. Then $E[\ell]$ is a free rank $1$ module over $\mathcal{O}/\ell\mathcal{O}$. The image $G$ of $\rho_{E,\ell}$ is contained in the $\mathcal{O}$-automorphism of $E[\ell]$, which are isomorphic to $(\mathcal{O}/\ell\mathcal{O})^{\times}$.
If $\ell$ is unramified in $\mathcal{O}$, then $G$ is contained in a group of order either $\ell^2-1$ or $(\ell-1)^2$. Since $\ell$ will not divide the group order of $G$, the action on $E[\ell]$ will be semi-simple.
Instead for ramified primes $\ell$, it can happen that $E[\ell]$ is not semi-simple. For instance for the curve $y^2 = x^3 + 1$ over $K=\mathbb{Q}(\sqrt{-3})$ and $\ell=3$. Then $K(E[3])/K$ is an extension of degree $3$ and hence the Galois group acts on $E[3]$ via the matrix $\bigr(\begin{smallmatrix} 1 & 1 \\ 0 & 1 \end{smallmatrix}\bigr)$.
I believe in the ramified case one has to check by hand if $E$ admits two isogenies of degree $\ell$ defined over $K$. Of course in practice one can determine $G$ completely for these $\ell$.