How many positive integer solutions does the equation x^2+y^2+z^2xzyz = 1 have? My guess is (1,0,1), (0,1,1) and (1,1,1). What is proof of that fact that there are none other?

1$\begingroup$ That's the equation of an ellipsoid: a bounded set. Diagonalizing it will give explicit bounds for $x$, $y$ and $z$. By diagonalization I mean a change of coordinates transforming it to the form $ax^2+by^2+cz^2=d$. $\endgroup$ – Robin Chapman May 30 '10 at 17:27

$\begingroup$ You may want to change "positive" to "nonnegative" in your question, and you may want to consider solutions with more zeroes. I'm afraid I'm voting to close, since your question is a little outside the scope of this site. Please see the FAQ for a list of problemsolving websites. $\endgroup$ – S. Carnahan♦ May 30 '10 at 18:20

$\begingroup$ Just to clarify before closing: write your equation (xz/2)^2 + (yz/2)^2 + z^2/2 =1 , whence z is either 0 or 1, &c. $\endgroup$ – Pietro Majer May 30 '10 at 18:30

$\begingroup$ Cindy, why don't you guess the fourth nonnegative solution $(0,0,1)$? (In my comment to Will's post below I explain how to give all rational points on your ellipsoid. It is this fourth solution through which the lines with rational pair of slopes pass.) $\endgroup$ – Wadim Zudilin Jun 2 '10 at 12:18
To elaborate on Robin's suggestion, set $x=v+w,y=u+w,z=u+v$, and the equation becomes $$u^2+v^2+2w^2=1,$$ with $u=(y+zx)/2,v=(xy+z)/2,w=(x+yz)/2$ being halfintegers. Now a brute force run through the possibilities is feasible, since $u\leq 1$ and such.
For the quick answer, you can use Mathematica:
Reduce[x x + y y + z z  x z  y z == 1, Integers]
or even wolframalpha:

$\begingroup$ Kevin, why does Mathematica prefer $xx$ instead of $x^2$? $\endgroup$ – Wadim Zudilin May 30 '10 at 23:38

1$\begingroup$ @Wadim: that's just me optimizing my typing. It's faster to type "x x" than "x^2". $\endgroup$ – Kevin O'Bryant May 31 '10 at 1:27

$\begingroup$ That's great, Kevin! I'll try to follow your trick (which I guess comes from ancient time). $\endgroup$ – Wadim Zudilin May 31 '10 at 3:02
Shame they are going to close this, I actually know something about this one. Your form is equivalent to $$ X^2 + Y^2 + Z^2 + Y Z + Z X + X Y $$ meaning there is an invertible integral change of variables. This is a regular form and integrally represents the same numbers as $$ U^2 + V^2 + 2 W^2 $$ which is to say all numbers not of shape $$ 4^k ( 16 m + 14 ) .$$ As it is positive definite, rather than just semidefinite, there are simple bounds on possible values for the variables in your original $$ x^2 + y^2 + z^2  y z  z x \leq M$$ which can be derived either by eigenvalues for the related symmetric Gram matrix,or, as I do, by Lagrange multipliers in maximizing $x^2$ or $y^2$ or $z^2$ subject to the constraint given. So all solutions to $$ x^2 + y^2 + z^2  y z  z x = M$$ can be found fairly quickly even for large $M.$
Well, see my article with Kaplansky and Schiemann,
http://zakuski.math.utsa.edu/~kap/Forms/Kap_Jagy_Schiemann_1997.pdf

$\begingroup$ Will, I am getting up too early. The quadratics correspond to the Cartan matrix which reminds me about RogersRamanujan identities. Apart from this, the OP sounds like homework. :) $\endgroup$ – Wadim Zudilin May 30 '10 at 21:44

$\begingroup$ Hi Wadim, it does give the appearance of homework as stated for the value $1$ on the right hand side. This would be a stretch for a schoolchild first exposed to "completing the square." I chose to ignore the $1$ and give the merest hint of what happens next. Meanwhile, I think you should go back to bed. Victor Wadimovich depends on you and you need your strength. $\endgroup$ – Will Jagy May 30 '10 at 22:11

$\begingroup$ Will, in spite of rain outside I have to teach, so no way to get a longer sleep. On my bus way to uni I wrote a general solution of the equation in rational (by intersecting the ellipsoid with the line $x=t(z1)$, $y=s(z1)$ for a rational pair of slopes $t,s$) to see that it does not give a simpler way to produce integer solutions. $\endgroup$ – Wadim Zudilin May 30 '10 at 23:43

$\begingroup$ Hello, Cindy. I have put a ton of related material at: zakuski.math.utsa.edu/~kap/forms.html On Wadim's idea, there should not be a really easy way to produce integral representations with arbitrary $M$ on the righthandside, see my answer (and the others) to mathoverflow.net/questions/3596 $\endgroup$ – Will Jagy May 31 '10 at 3:04