EDIT This answer doesn't satisfy OP's assumptions. Currently keeping it because of existence of infinitely many solutions over extensions of the integers and the relation with N-conjecture.
If you set $m=n$ you have infinitely many solutions $x=y$.
If $m=n=2$, the solutions are $x=y,x= -y-1$.
If $m=n=3$ we have the factorization $(-x + y) \cdot (x^2 + x*y + y^2 + x + y)$. Maybe for deep reasons, the quadratic factor doesn't have solutions over the integers, but we believe it has infinitely many solutions over $\mathbb{Z}[i]$.
For $m=n=4$ we have a cubic factor, which is expected to be of genus $1$, and according to sage it is genus zero, which might give integral points.
For $ m=n < 30$, the higher degree factor is genus zero, probably some algebraic geometer will explain it.