On Markoff-type diophantine equation

Question

Do there exist integers $x,y,z$ such that $$ x^2+y^2-z^2 = xyz -2 \quad ? $$

Why this is interesting? First, this equation arose in an answer to the previous Mathoverflow question What is the smallest unsolved diophantine equation? but was not asked explicitly as a separate question. The context is that, in a well-defined sense for the notion of "smallness", the equation above is the "smallest" open Diophantine equation.

Second, this equation is one of the simplest non-trivial representative of the family of equations $ax^2+by^2+cz^2=dxyz+e$, which generalises a well-known Markoff equation $x^2+y^2+z^2=3xyz$. The well-known methods for the former (Vieta jumping) has been extended to the general case if $a,b,c$ are all natural numbers and are divisors of $d$ (see, for example, Fine, Benjamin, et al. "On the Generalized Hurwitz Equation and the Baragar–Umeda Equation." Results in Mathematics 69.1-2 (2016): 69-92). The question seems to be much more challenging when $a,b,c$ have different signs. The simplest case with different signs is $a=b=d=1$ and $c=-1$, which leads to the family of equations $x^2+y^2-z^2=xyz+e$. The equation above is the first non-trivial example from this family.

Answer 1

There is no solution.

Fix a solution $(x,y,z)$ with $|x|+|y|+|z|$ minimal. We will show a contradiction.

We can't have $xyz=0$ as we would then obtain one of the unsolvable equations $x^2+y^2= -2$, $x^2-z^2=-2$, $y^2-z^2=-2$.

If $xyz>0$, then by swapping the signs of two of $x,y,z$ if necessary we can assume $x,y,z>0$, and switching $x$ and $y$ we can assume $x \geq y$. We have the Vieta jump $x \to yz-x$, so if this is minimal we have $x \leq yz/2$. Since $f(x)=x^2+y^2-z^2 - xyz + 2 $ is convex and vanishes at $x$, we must $f(y) \geq 0$ or $f(yz/2) \geq 0$.

But $$f(y)= (2-z)y^2 -z^2 + 2$$ so $f(y) \leq 0$ imply $z<2$ and $z=1$ gives the impossible $x^2+y^2-xy=-1$ and $$f(yz/2) = y^2 -z^2 - y^2 z^2/4 +1= (1-z^2/4)y^2 - z^2 +1$$ which again is nonnegative only if $z<2$ which is impossible.

If$xyz<0$, then by swapping the signs if necessary we can assume $x,y,z<0$. We have the Vieta jump $z \to -xy-z$, so if this is minimal we have $z \geq -xy/2$. We have $g(z)=z^2 + xyz - x^2-y^2-2$ is convex and vanishes at $z$, we must have $g(0) \geq 0$ or $g(-xy/2) \geq 0$.

But $g(0) = -x^2 - y^2 - 2 <0$ and $g(-xy/2) = - x^2 y^2/4 - x^2 - y^2 -2 <0$.

So neither case is possible.