(Essentially the same argument as the one given by Will Sawin, but perhaps a bit simpler.)

If $A$ is an abelian variety over a field $k\supset\mathbb{Q}$, then the tangent space $T_0(A)$ at identity is a module over $\mathrm{End}_{k}(A)\otimes\mathbb{Q}$.

Now, if the latter contains a field $K$, then $T_0(A)$ has to have dimension at least 1 over $K$. On the other hand $T_0(A)$ has dimension $\dim_k(A)$ over $k$.

(Essentially the same argument as the one given by Will Sawin, but perhaps a bit simpler. Further clarification included thanks to comment by Wojowu.)

If $A$ is an abelian variety over a field $k\supset\mathbb{Q}$, then the tangent space $T_0(A)$ at identity is a module over $\mathrm{End}_{k}(A)\otimes\mathbb{Q}$.

Now, if the latter contains a field $K$, then $T_0(A)$ has to have dimension at least 1 over $K$. On the other hand $T_0(A)$ has dimension $\dim(A)$ over $k$. Thus $[K:\mathbb{Q}]\leq\dim(A)$.

Added: Alternate explanation of above.

Consider $A(\mathbb{R})$ as a Lie group with connected component $A(\mathbb{R})_0$. The exponential map $T_0(A)\to A(\mathbb{R})_0$ is the universal covering of a compact torus of real dimension $n=\dim(A)$. It is clear that elements of $\mathrm{End}_{\mathbb{R}}(A)$ lift to this cover; Let $K$ be a subfield of $\mathrm{End}_{\mathbb{R}}(A)$. Note that $\mathcal{O}_K$ is a domain and the covering group (which is $\mathbb{Z}^{n}$) is a module over $\mathcal{O}_K$. Thus the rank of $\mathcal{O}_k$ as a $\mathbb{Z}$ module is at most $n$.