Stack Exchange network consists of 183 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.
If you consider the iteration proposed involving $T(U)$, and let $U_0=I$, then the eigenvalues of $AB$ are log majorized by the eigenvalues of $AU_1B$. This is because the $AB$ and $AU_1B$ are similar. Unfortunately, I don't see how this can be used to prove the desired inequality.
Was the motivation for the inequality the operator $T(U)=AU^*B$. This inequality must have something to do with doing the power method with this operator and normalizing using a polar factorization. $$T(U_i)=U_{i+1}^*P_{i+1}$$
I am generating $U$ in the manner suggested by the discussion above. We want $$AUB=BU^*A$$ so assume that $AU=BH$ where $H$ is a Hermitian matrix. Then $A^2=BH^2B$, and I get $H$ from taking the positive square root of $B^{-1}A^2B^{-1}$, and then $U=A^{-1}BH$. As the discussion above suggests, different roots could be used for generating $H$. And having thought about this a little more, I would say the resulting iterations seems more like a modified Duggal iteration.
I don't have an answer nor do I have the reputation to comment on the previous comments, but it occurred to me to look at what happens when you iterate this operation. That is given A and B construct $U$, let $A=U^*AU$ and repeat. It seems that going in the forward direction, eventually the matrices $A$ and $B$ commute with the eigenvalues of $A$ matched with the eigenvalues of $B$ so that largest is paired with largest. If you run the iteration in reverse, again they eventually commute but at the end the eigenvalues are paired sorted so that the largest of $A$ is paired with smallest of $B$ a