Let $F$ be a number field and $A$ an abelian variety over $F$. It is known that if $A$ has complex multiplication, then it has potentially good reduction everywhere, namely there exists a finite extension $L$ of $F$ such that $A_L$ has good reduction over every prime of $L$.

And what about the inverse: if $A$ is known to be of potential good reduction everywhere, how far is it from having complex multiplication?

As the reduction behavior is determined by the Galois representations of the decompositon groups, one can reformulate the problem as follows: let $A$ be an abelian variety over $F$, $p$ a fixed rational prime, $V$ the p-adic Tate module of $A$; and for $\lambda$ primes of $F$, $\rho_\lambda$ is the $p$-adic representation on $V$ of the decomposition group $G_\lambda$ at $\lambda$. If $\rho_\lambda$ is potentially unramified for $\lambda$ not dividing $p$, and potentially cristalline for $\lambda$ dividing $p$, do we know that the global Galois representation $\rho$ on $V$ is potentially abelian, i.e. when shifting to some open subgroup, the image of $\rho$ is contained in a torus of $GL_{\mathbb{Q}_p}(V)$? What do we know about the Fontaine-Mazur conjecture in this case?

Thanks!