I would like to know if for $A,B\in SO(3)$ the inequality $$ \|AB-BA\|_F\leq \|A-I\|_F\|B-I\|_F $$ holds, where $\|\cdot\|_F$ denotes the Frobenius norm and $I$ the identity matrix. Using the identity $$ AB-BA=(A-I)(B-I)-(B-I)(A-I) $$ one can show the inequality with a factor $2$. However the inequality seems to be true also without this factor.
I encountered this problem when I tried to estimate the error of permuting two small rotations.
I confirm Peter's suggestion that the best constant is $\frac1{\sqrt2}$. For let $\omega$ be this best constant. Then the inequality amounts to writing $$3-{\rm Tr}(A^tB^tAB)\le2\omega^2(3-{\rm Tr}\,A)(3-{\rm Tr}\,B).$$ Let me parametrize $A$ by the angle of rotation $\theta$ and the axis of rotation $u$, a unit vector. Likewise $B$ has an angle of rotation $\alpha$ and an axis $v$. The left-hand side above is $8\omega^2(1-c)(1-c')$, where $c=\cos\theta$ and $c'=\cos\alpha$.
Now $B^tAB$ is the rotation of same angle $\theta$ about $w=B^{-1}u$, and $A^t$ the rotation of angle $-\theta$ about $u$. It is an instructive calculus that $${\rm Tr}(A^tB^tAB)=2c+c^2+(1-c)^2(w\cdot u)^2+2s^2w\cdot u.$$ Because of $w\cdot u\ge c'$, we find that $\omega$ is the best constant in $$3-2c-c^2+(1-c)^2c'^2-2s^2c'\le8\omega^2(1-c)(1-c').$$ The left-hand side factorizes as $(1-c)(1-c')(3+c+c'-cc')$, and there remains the inequality $$3+c+c'-cc'\le8\omega^2.$$ Therefore $\omega^2$ is the supremum of $\frac18(3+c+c'-cc')$ over $[-1,1]^2$. Because this quantity if affine in both $c$ and $c'$, the supremum is achieved at a vertex, and its value is $4$. Whence $\omega^2=\frac48=\frac12$.
Numerical experiments suggest that actually $\|AB-BA\|_F\leq\frac{1}{\sqrt{2}} \|A-I\|_F\|B-I\|_F$ could be true, and that this bound is sharp.
I don't know if that helps, but this sharpened inequality is equivalent to \begin{equation} 3-\text{trace}(ABA^tB^t)\le(3-\text{trace}(A))(3-\text{trace}(B)). \end{equation}
