# matrix inequality with orthogonal matrices

I would like to know if for $A,B\in SO(3)$ the inequality $$\|AB-BA\|_F\leq \|A-I\|_F\|B-I\|_F$$ holds, where $\|\cdot\|_F$ denotes the Frobenius norm and $I$ the identity matrix. Using the identity $$AB-BA=(A-I)(B-I)-(B-I)(A-I)$$ one can show the inequality with a factor $2$. However the inequality seems to be true also without this factor.

I encountered this problem when I tried to estimate the error of permuting two small rotations.

• The inequality holds with constant $\sqrt{2}$ for arbitrary $3\times 3$ matrices, see Bottcher, Wenzel, The Frobenius norm of the commutator, 2008. – Christian Remling Nov 9 '16 at 22:21

I confirm Peter's suggestion that the best constant is $\frac1{\sqrt2}$. For let $\omega$ be this best constant. Then the inequality amounts to writing $$3-{\rm Tr}(A^tB^tAB)\le2\omega^2(3-{\rm Tr}\,A)(3-{\rm Tr}\,B).$$ Let me parametrize $A$ by the angle of rotation $\theta$ and the axis of rotation $u$, a unit vector. Likewise $B$ has an angle of rotation $\alpha$ and an axis $v$. The left-hand side above is $8\omega^2(1-c)(1-c')$, where $c=\cos\theta$ and $c'=\cos\alpha$.
Now $B^tAB$ is the rotation of same angle $\theta$ about $w=B^{-1}u$, and $A^t$ the rotation of angle $-\theta$ about $u$. It is an instructive calculus that $${\rm Tr}(A^tB^tAB)=2c+c^2+(1-c)^2(w\cdot u)^2+2s^2w\cdot u.$$ Because of $w\cdot u\ge c'$, we find that $\omega$ is the best constant in $$3-2c-c^2+(1-c)^2c'^2-2s^2c'\le8\omega^2(1-c)(1-c').$$ The left-hand side factorizes as $(1-c)(1-c')(3+c+c'-cc')$, and there remains the inequality $$3+c+c'-cc'\le8\omega^2.$$ Therefore $\omega^2$ is the supremum of $\frac18(3+c+c'-cc')$ over $[-1,1]^2$. Because this quantity if affine in both $c$ and $c'$, the supremum is achieved at a vertex, and its value is $4$. Whence $\omega^2=\frac48=\frac12$.
Numerical experiments suggest that actually $\|AB-BA\|_F\leq\frac{1}{\sqrt{2}} \|A-I\|_F\|B-I\|_F$ could be true, and that this bound is sharp.
I don't know if that helps, but this sharpened inequality is equivalent to $$3-\text{trace}(ABA^tB^t)\le(3-\text{trace}(A))(3-\text{trace}(B)).$$
• Maybe the last inequality translates to an easier statement for ${\rm SU}(2,\mathbb{C}).$ – Geoff Robinson Nov 10 '16 at 13:04
• I think in fact for $n\times n$ matrices, we could get $1/\sqrt{n-1}$ as the multiplicative factor...does that match your simulations? – Suvrit Nov 10 '16 at 15:16