Here is an alternative formulation (possibly your original one) where
$x_m$ is replaced by $n +$ something which yields $0 < i+j+k$ with each of
$i,j,k \ge -n$ . Then (I've already fixed one mistake, so check my work)

$2(i+j+k+1)n + (ij+jk +ki) =  a$

$(i+j+k)n^2 + (ij+jk+ki)n +ijk = an - b$

$(i+j+k+2)n^2  - ijk = b$

Since $(ij +jk +ki)$ can be negative, we don't have $a > n$ or even $b> 0$.

However there are inequalities mentioned in other posts which apply to
the terms $(s-1) = (i+j+k)$ and $t =(ij +jk +ki)$.  <strike>Further, one has
$an/2  - b = ijk $.</strike>  So it might be useful to rewrite the system using
$s$ and $t$ and solve it
 given $n$, and then see if $i,j,k$ can be found after that.

Gerhard "Ask Me About System Design" Paseman, 2011.02.16