This is not true in general. Take your example and reembed everything using the $n$-th Veronese map. A subvariety $F$ of $X$ is reembedded as a variety of degree ${\rm deg}(F) n^{\dim(F)}$. In particular, bigger dimensional fibers will have larger degree than smaller dimensional fibers, provided $n$ is large enough.
EDIT: This is what I had in mind. Choose a positive integer $d$. Let $Z=(z_1,\ldots,z_N)$ be a list of all the monomials of degree at most $d$ in $x_1,\ldots,x_n$ such that $(z_1,\ldots,z_M)$ is a list of all the monomials of degree at most $d$ in $x_1,\ldots,x_m$. Using the monomials $Z$ we obtain an embedding $v_d \colon X \to \mathbb{A}^N$, called the $d$-th Veronese embedding. A property of $v_d$ is that for every subvariety $F$ of $\mathbb{A}^n$ the equality $deg(v_d(F)) = deg(F) \, d^{dim(F)}$ holds. Let $\pi_d \colon X \to \mathbb{A}^M$ be the projection on the first $M$ coordinates. The morphism $\pi_d$ is identical to the morphism $\pi$ (whatever that means), but the fibers of $\pi_d$ as subschemes of $\mathbb{A}^N$ are isomorphic to the image under the $d$-th Veronese embedding of the fibers of $\pi$. Thus, for $d$ large enough, the fibers of larger dimension will have degree larger than the fibers of smaller dimension.
This is not true in general. Take your example and reembed everything using the $n$-th Veronese map. A subvariety $F$ of $X$ is reembedded as a variety of degree ${\rm deg}(F) n^{\dim(F)}$. In particular, bigger dimensional fibers will have larger degree than smaller dimensional fibers, provided $n$ is large enough.