Here is an alternative formulation (possibly your original one) where $x_m$ is replaced by $n +$ something which yields $0 < i+j+k$ with each of $i,j,k \ge -n$ . Then (I've already fixed one mistake, so check my work)
$2(i+j+k+1)m 2(i+j+k+1)n + 2(ij+jk (ij+jk +ki) = a$
$(i+j+k)n^2 + (ij+jk+ki)n +ijk = an - b$
$(i+j+k+1)n^2 +( ij+jk+ki)n (i+j+k+2)n^2 - ijk = b$
Since $(ij +jk +ki)$ can be negative, we don't have $a > n$ or even $b> 0$.
However there are inequalities mentioned in other posts which apply to
the terms $(s-1) = (i+j+k)$ and $t =(ij +jk +ki)$.
Further, one has
$an/2 - b = ijk $. So it might be useful to rewrite the system using
$s$ and $t$ and solve it
given $n$, and then see if $i,j,k$ can be found after that.
Gerhard "Ask Me About System Design" Paseman, 2011.02.16