On Markoff-type diophantine equation

Do there exist integers $$x,y,z$$ such that $$x^2+y^2-z^2 = xyz -2 \quad ?$$

Why this is interesting? First, this equation arose in an answer to the previous Mathoverflow question What is the smallest unsolved diophantine equation? but was not asked explicitly as a separate question. The context is that, in a well-defined sense for the notion of "smallness", the equation above is the "smallest" open Diophantine equation.

Second, this equation is one of the simplest non-trivial representative of the family of equations $$ax^2+by^2+cz^2=dxyz+e$$, which generalises a well-known Markoff equation $$x^2+y^2+z^2=3xyz$$. The well-known methods for the former (Vieta jumping) has been extended to the general case if $$a,b,c$$ are all natural numbers and are divisors of $$d$$ (see, for example, Fine, Benjamin, et al. "On the Generalized Hurwitz Equation and the Baragar–Umeda Equation." Results in Mathematics 69.1-2 (2016): 69-92). The question seems to be much more challenging when $$a,b,c$$ have different signs. The simplest case with different signs is $$a=b=d=1$$ and $$c=-1$$, which leads to the family of equations $$x^2+y^2-z^2=xyz+e$$. The equation above is the first non-trivial example from this family.

There is no solution.

Fix a solution $$(x,y,z)$$ with $$|x|+|y|+|z|$$ minimal. We will show a contradiction.

We can't have $$xyz=0$$ as we would then obtain one of the unsolvable equations $$x^2+y^2= -2$$, $$x^2-z^2=-2$$, $$y^2-z^2=-2$$.

If $$xyz>0$$, then by swapping the signs of two of $$x,y,z$$ if necessary we can assume $$x,y,z>0$$, and switching $$x$$ and $$y$$ we can assume $$x \geq y$$. We have the Vieta jump $$x \to yz-x$$, so if this is minimal we have $$x \leq yz/2$$. Since $$f(x)=x^2+y^2-z^2 - xyz + 2$$ is convex and vanishes at $$x$$, we must $$f(y) \geq 0$$ or $$f(yz/2) \geq 0$$.

But $$f(y)= (2-z)y^2 -z^2 + 2$$ so $$f(y) \leq 0$$ imply $$z<2$$ and $$z=1$$ gives the impossible $$x^2+y^2-xy=-1$$ and $$f(yz/2) = y^2 -z^2 - y^2 z^2/4 +1= (1-z^2/4)y^2 - z^2 +1$$ which again is nonnegative only if $$z<2$$ which is impossible.

If$$xyz<0$$, then by swapping the signs if necessary we can assume $$x,y,z<0$$. We have the Vieta jump $$z \to -xy-z$$, so if this is minimal we have $$z \geq -xy/2$$. We have $$g(z)=z^2 + xyz - x^2-y^2-2$$ is convex and vanishes at $$z$$, we must have $$g(0) \geq 0$$ or $$g(-xy/2) \geq 0$$.

But $$g(0) = -x^2 - y^2 - 2 <0$$ and $$g(-xy/2) = - x^2 y^2/4 - x^2 - y^2 -2 <0$$.

So neither case is possible.

• This is matter of taste, but I prefer testing Vieta jumps by product (rather than sum) Vieta theorem. Here we see that if $|x|=\max(|x|,|y|,|z|)$, then the second root $\tilde{x}$ w.r.t. to $x$ equals $\tilde{x}=(y^2+2-z^2)/x$ and $|\tilde{x}|<|x|$, we may perform a jump. If $|z|=\max(|x|,|y|,|z|)$, by changing signs assume that $x,y>0$, the second root w.r.t. $z$ satisfies $\tilde{z}z=-x^2-y^2-2<0$. If $z<0$, then $\tilde{z}=-xy-z<-z$ and so $|\tilde{z}|=\tilde{z}<|z|$, do a jump. Finally if $z>\max(x,y)$, then $z^2+xyz\ge (x+1)^2+y^2>x^2+y^2+2$. May 17, 2021 at 13:39