# Invertible matrices satisfying $[x,y,y]=x$.

I have been thinking about this question for quite some time but now this question by Denis Serre revived some hope.

Question. Let $x,y$ be invertible matrices (say, over $\mathbb C$) and $[x,y,y]=x$ where $[a,b]=a^{-1}b^{-1}ab$, $[a,b,c]=[[a,b],c]$. Does it follow that some power of $x$ is unipotent?

The motivation is this. Consider the one-relator group $\langle x,y \mid [x,y,y]=x\rangle$. It is hyperbolic (proved by A. Minasyan) and residually finite (that is proved in my paper with A. Borisov). If the answer to the above question is "yes", then that group would be non-linear which would provide an explicit example of non-linear hyperbolic group.

Update 1. Can $x$ in the above be a diagonal matrix and not a root of 1?

Update 2. The group is residually finite, so it has many representations by matrices such that $x, y$ have finite orders (hence their powers are unipotents).

Update 3. The group has presentation as an ascending HNN extension of the free group: $\langle a,b,t \mid a^t=ab, b^t=ba\rangle$. So it is related to the Morse-Thue map. Properties of that map may have something to do with the question. See two quasi-motivations of the question as my comments below.

-
Are there any one-relator groups known not to be linear? – Łukasz Grabowski Nov 3 '10 at 23:06
@Lukasz: Yes, there are even non-residually finite ones: $BS(2,3)=\langle x,y \mid y^{-1}x^2y=x^3\rangle$. There are also residually finite 1-related groups which are not linear. Those were constructed in our paper with Cornelia Drutu (in J. Algebra). The point is that this group is hyperbolic. There is an example of a non-linear hyperbolic group due to M. Kapovich (which easily follows from the super-rigidity of certain rank 1 lattices and a Gromov-Olshanskii theorem). But that example has no explicit presentation. This one would be the first explicit example. – Mark Sapir Nov 3 '10 at 23:12
Here is one of the quasi-reasons why I think the answer is positive. If $G=\langle x,y \mid [x,y,y]=x\rangle$ is linear, then it has a representation over a number field, hence over $\mathbb{Q}$. Therefore the sequence of indexes of subgroups of finite index of $G$ must grow polynomially (take congruence subgroups). This would imply that certain polynomial maps over finite fields have many quasi-fixed points with long orbits (see our paper with Borisov). The latter seems to be impossible. – Mark Sapir Mar 18 '12 at 14:04
A trivial observation: setting $z := [x,y]$, the condition $[x,y,y]=x$ is equivalent to the assertion that the pair $(x,z)$ is conjugate to $(xz,zx)$ after conjugation by $y$. So the question is equivalent to the question of whether a pair of matrices $(x,z)$ which has the property of being conjugate to $(xz,zx)$ is such that all the eigenvalues of $x$ (or equivalently, $z$, which is necessarily conjugate to $x$) are roots of unity. Unfortunately, I got stuck after this observation: the conjugacy does give a number of trace identities involving various words in z,x, but not enough of them... – Terry Tao Mar 18 '12 at 19:40
@Terry: Yes, this was another quasi-reason. Consider $G=\langle x,y\mid x^y=x^2\rangle$. Then in every linear representation of $G$, conjugating $x$ by powers of $y^{-1}$ will produce matrices that are closer and closer to 1. So if $x,y$ are matrices $\lim_{n\to\infty} x^{y^{-n}}=1$. This means that $x$ is a unipotent element "of $y$" in Margulis' terminology, hence $x$ is unipotent. Now we have a similar presentation $\langle x,z,y\mid x^y=xz, z^y=zx\rangle$, so the idea was to show that some power of $x$ satisfies the limit property above. – Mark Sapir Mar 18 '12 at 21:02

Take $n$ to be small: Try $2$ first, and $5$ is probably near the limit of a computer algebra system. Choose $x$ to be a random $n \times n$ diagonal matrix with determinant $1$, for example, $\mathrm{diag}(17, 1/17)$. Write out your relation, leaving all the elements of $y$ as variables. After clearing denominators, you have $n^2$ simultaneuous homogenous equations in $n^2$ variables. (If I haven't made any dumb errors, they have degree $3n$.) Ask your favorite computer algebra system to solve them for you. If any of the roots are not on the hypersurface $\det y=0$, then you have a counterexample!
I did it for $n=2$, of course. The conjecture is true in that case. For $n=2$ you can use the trace identities. That reduces dimension to 3 (every pair of matrices is determined by three traces, if I remember correctly). It is written in the paper with Drutu which I mentioned above. For other $n$'s I did not check. There are no trace identities and the computation is too large. – Mark Sapir Nov 4 '10 at 1:37
@David: Just to clarify my previous comment. Every pair of $2\times 2$- invertible matrices of det 1 is determined by the traces $tr(a), tr(b), tr(ab)$ up to conjugacy. There are polynomial identities allowing to compute the trace of the word $tr(w(a,b))$ if you know $tr(a), tr(b), tr(ab)$. Then the relation $[x,y,y]x^{-1}=1$ gives that certain trace is equal to 2, etc. – Mark Sapir Nov 4 '10 at 2:23
OK, got it. Yeah, trace identities would be the way to do this for $n=2$, and maybe for $n=3$. I think just writing out the equations should win for $n=4$, though I haven't tried it. But my point was just that you should be doing these basic low dimensional checks, and it sounds like you are. – David Speyer Nov 4 '10 at 2:59