I still don't have enough reputations points to comment but here is another related iteration. Geometrically the inequality says that using the matrix $U$ to transform $A$ moves the hyperellipse associated with $A$ further away from the hyperellipse associated with $B$. This has something to do with the angles between the eigenvectors of $A$, $U^*AU$ and $B$.

If we diagonalize $A=V_A\Lambda_AV_A^*$ and $B=V_B\Lambda_BV_B^*,$
then $$AUB=V_A\Lambda_A V_A^*UV_B\Lambda_B V_B^*$$. Taking a similarity transformation with $V_A\Lambda_A$ and letting $\Theta_2=V_A^*UV_B$ and $\Theta_1=V_A^*V_B$ denote the matrices whose entries have the information regarding the angles between the eigenvectors of $A$ , $U^*AU$ and $B$, then we have
$$H_1=\Theta_2\Lambda_B \Theta_1^*\Lambda_A^{-1} $$
which is a Hermitian matrix. We have
$$ H_1\Lambda_A\Theta_1=\Theta_2\Lambda_B$$
At the next iteration we would have
$$ H_2\Lambda_A\Theta_2=\Theta_3\Lambda_B$$
Eliminating $\Theta_3$, we have
$$ H_2^2=\Lambda_A^{-1}\Theta_2\Lambda_B^2\Theta_2^*\Lambda_A^{-1},$$
and now eliminating $\Lambda_B$ and $\Theta_2$ and taking a square root,
$$H_2=(\Lambda_A^{-1}H_1\Lambda_A^2H_1\Lambda_A^{-1})^{1/2}.$$
$\Lambda_B$ is absent from this iteration though it can be recovered. It seems that the 2-norm of the $H_i$ is increasing and that $H_i$ must converge to $\Lambda_B\Lambda_A^{-1}$ where the eigenvalues of $A$ are in ascending order and the eigenvalues of $B$ are in descending order.

19more comments