This might be a very simple question, and that might be the reason that I could not find any reference on this.

My question is

Let $A$ be an abelian variety defined over a number field $k$, and $N$ the conductor. Let $m\geq 2$. Consider the division field $k(A[m])$. Let $\mathfrak{p}$ be a prime ideal in $k$ that divides $m$. Also suppose that $k(A[m])\neq k$. Then is it true that $$\mathfrak{p} \textrm{ is ramified in } k(A[m]) ? $$

If this is not true, then can anyone provide a counterexample?

I know that if $\mathfrak{p}$ does not divide $mN$, then it should be unramified.