MathOverflow will be down for maintenance for approximately 3 hours, starting Monday evening (06/24/2013) at approximately 9:00 PM Eastern time (UTC-4).

Orthogonal similarity of matrices

Given $M\in M_n({\mathbb R})$ and $\ell\in{0,\ldots,n-1}$, we define $$d_\ell(M)=\sum_{j=1}^nm_{j,j+\ell},$$ where the indices are understood mod $n$. In particular, $d_0$ is the trace operator.

Let $A\in M_n({\mathbb R})$ be given. We define a map $\Delta: O_n({\mathbb R})\rightarrow{\mathbb R}^{n-1}$ by $$\Delta(R)=(d_1(R^TAR),\ldots,d_{n-1}(R^TAR)).$$ Mind that we omit $d_0(R^TAR)$, because we know in advance that it equals the trace of $A$.

Question. Does it exist an orthogonal matrix $R$ such that $\Delta(R)=(0,\ldots,0)$ ?

The requested property ressembles one for which the answer is known to be positive: find $R$ orthogonal such that the diagonal $R^TAR$ is constant (thus equal to $\frac{1}{n}{\rm Tr}A$). Both properties consist of $n-1$ linear constraints, and both are consistent with the fact that the mean value of $R^TAR$ over $SO_n$ is $(\frac{1}{n}{\rm Tr}A) I_n$. Thus the answer would certainly be positive if the stronger following statement is true.

Statement. The image of $SO_n$ under $\Delta$ is convex. True or False ?

This statement looks ambitious, since $\Delta$ is not linear, and $SO_n$ is not a convex set. But an optimistic mathematicien will say that it ressembles the Toeplitz-Hausdorff theorem about the convexity of the image of the complex unit sphere under the quadratic map $x\mapsto x^*Mx$. Note that the T-H thm is used to find an $R^TAR$ with constant diagonal.

-
Just to understand better your question: if we work with complex matrices, $A$ is unitarily equivalent to a lower triangular matrix, so the question becomes trivial, right? is there any relation with your problem? – Piero D'Ancona Sep 29 2010 at 15:13
No, it is not trivial. Let me emphasize the indices are understood mod n. Thus the functions $d_\ell$ are circular sums. There are $n$ terms in each sum. – Denis Serre Sep 29 2010 at 15:37
I see. Not trivial. – Piero D'Ancona Sep 29 2010 at 16:15
For this question, I would have suggested to go and see a certain very recent book on matrices... but will not, as it's your book ;-) – Pietro Majer Sep 29 2010 at 16:20
If you took $R$ to be unitary, then I think it is equivalent to the other problem, by diagonalizing the cyclic permutation matrix. – Ian Agol Sep 30 2010 at 5:52
show 1 more comment

Believe me, I didn't know the answer when I asked the question. But now I do. It is No. Here is a counterexample, a $3\times3$ matrix $A$ for which $\Delta_A$ does not vanish over the orthogonal group.
The matrix is that of a rotation of angle $2\pi/3$ around some axis. For instance $A$ can be taken as the matrix of the permutation $[1,2,3]$. Its orbit under orthogonal conjugation is the set of all rotations of angle $2\pi/3$. So let $B$ be such a rotation, and let $v=(a,b,c)$ be the unitary vector about which the rotation takes place. Then $d_1(B)$ and $d_2(B)$ are $$\frac{3}{2}(ab+bc+ca)\pm\frac{\sqrt3}{2}(a+b+c).$$ Thus $d_1=d_2=0$ means $ab+bc+ca=0$ and $a+b+c=0$, which are incompatible with $a^2+b^2+c^2=1$.