(Essentially the same argument as the one given by Will Sawin, but perhaps a bit simpler.)

If $A$ is an abelian variety over a field $k\supset\mathbb{Q}$, then the tangent space $T_0(A)$ at identity is a module over $\mathrm{End}_{k}(A)\otimes\mathbb{Q}$.

Now, if the latter contains a field $K$, then $T_0(A)$ has to have dimension at least 1 over $K$. On the other hand $T_0(A)$ has dimension $\dim_k(A)$ over $k$.