Given two $N \times N$ symmetric matrices $A, B$, where $A$ is positive semidefinite while $B$ is not positive semidefinite. I am interested in solving unitary constrained trace maximization problem:
$\arg \max_{U}\text{trace}(UAU^TB),\;\;$ subject to $\;U^TU=I.$
When the matrix $B$ is positive semidefinite and diagonal the solution set $U$ will be the ordered eigenvectors of $A$. But in my case the matrix $B$ is surely nonpositive definite.
$\DeclareMathOperator{tr}{tr}$ $\DeclareMathOperator{grad}{grad}$ $\DeclareMathOperator{sym}{sym}$
New answer:
WLOG, We can assume that $B$ is positive definite. Since for every orthogonal matrix $U$, we have
$$ \tr(UAU^T(B+\lambda I)) = \tr(UAU^TB) + \lambda \tr(A)$$
So if we use $B+\lambda I$ (for sufficiently large $\lambda$) instead of $B$, the optimal solution set were not changed.
Old answer:
Let us, first, find critical points of the problem. One can see the problem as an unconstrained optimization problem on the manifold of orthogonal matrices. So first order optimality conditions are $\grad f(U) = 0$, where $\grad$ stand for the Riemannian gradient of $f(U) := \tr(UAU^TB)$ on the orthogonal matrices. We have $\grad f(U) = 2AU^TB-2U^T \sym(UAU^TB)$, where $\sym$ is the symmetric part of a matrix. Now, necessary optimality conditions reduce to $B$ commutes with $UAU^T$. So when $B$ is a diagonal matrix (not necessarily semi-positive definite) with distinct elements, $UAU^T$ is diagonal and columns of $U^T$ are eigenvectors of $A$.
For Hermitian matrices $A$ and $B$, we know that the following inequality holds: \begin{equation*} \text{tr}(AB) \le \langle \lambda^{\downarrow}(A), \lambda^{\downarrow}(B)\rangle, \end{equation*} where $\lambda^{\downarrow}(\cdot)$ denotes the eigenvalues arranged in decreasing order. Using this inequality we see that to maximize the innerproduct, $U^T$ should be the eigenvectors of $A$ (suitably ordered).
