Here is an alternative formulation (possibly your original one) where $x_m$ is replaced by $n +$ something which yields $0 < i+j+k$ with each of $i,j,k \ge -n$ . Then

$2(i+j+k+1)m + 2(ij+jk +ki) = a$

$(i+j+k)n^2 + (ij+jk+ki)n +ijk = an - b$

$(i+j+k+1)n^2 +( ij+jk+ki)n - ijk = b$

Since $(ij +jk +ki)$ can be negative, we don't have $a > n$ or even $b> 0$.

However there are inequalities mentioned in other posts which apply to the terms $(s-1) = (i+j+k)$ and $t =(ij +jk +ki)$. Further, one has $an/2 - b = ijk $. So it might be useful to rewrite the system using $s$ and $t$ and solve it given $n$, and then see if $i,j,k$ can be found after that.

Gerhard "Ask Me About System Design" Paseman, 2011.02.16

Here is an alternative formulation (possibly your original one) where $x_m$ is replaced by $n +$ something which yields $0 < i+j+k$ with each of $i,j,k \ge -n$ . Then (I've already fixed one mistake, so check my work)

$2(i+j+k+1)n + (ij+jk +ki) = a$

$(i+j+k)n^2 + (ij+jk+ki)n +ijk = an - b$

$(i+j+k+2)n^2 - ijk = b$

Since $(ij +jk +ki)$ can be negative, we don't have $a > n$ or even $b> 0$.

However there are inequalities mentioned in other posts which apply to the terms $(s-1) = (i+j+k)$ and $t =(ij +jk +ki)$. Further, one has $an/2 - b = ijk $. So it might be useful to rewrite the system using $s$ and $t$ and solve it given $n$, and then see if $i,j,k$ can be found after that.

Gerhard "Ask Me About System Design" Paseman, 2011.02.16