In the paper "Points of Order 13 on Elliptic Curves" by Mazur-Tate, they say in the introduction:
It seemed ... that if such an abelian variety $J$, which has bad reduction at only one prime and has a sizeable number of endomorphisms, has a point of order $19$, it is not entitled to have any other points.
where $J$ is the Jacobian of $X_1(13)$. Can this heuristic be made more precise? They go on to prove in the paper that $J$ has exactly $19$ rational points but I am more interested in the intuition behind the statement.