Yes. This follows from the main result of the following paper of Zarhin.
MR0885780 (88h:14046) Zarhin, Yu. G.(2-AOS-CN) G. Endomorphisms and torsion of abelian varieties. Duke Math. J. 54 (1987), no. 1, 131–145.
His result, specialized to the $K$-simple case, is the following (fantastic) theorem: let .
Let
$A$ be a $K$-simple abelian variety defined over a number field $K$. TFAEThe following are equivalent:
(i) $A(K^{\operatorname{ab}})[\operatorname{tors}]$ is infinite.
(ii) $A$ is of CM-type over $K$.
Your hypotheses imply that there is infinite torsion over the abelian extension cut out by the action of Galois on the one-dimensional subspace (the Galois group is contained in $\mathbb{Z}_{p}^{\times}$), so Zarhin's theorem applies.

