Find the transformation $P$ that minimizes the following:

Question

$$\displaystyle\min_{\mathbf{P}} \text{trace}(\mathbf{APP^HA^H}) \quad{} \text{subject to} \quad{} \text{trace}(\mathbf{(I-P)(I-P)^H})=\alpha, \alpha \geq0$$ Can also be rewritten as $$\displaystyle\min_{\mathbf{P}} \text{trace}(\mathbf{APP^HA^H}) \quad{} \text{subject to} \quad{} \mathbf{\|I-P\|}_F^2=\alpha, \alpha \geq0$$ where $\mathbf{I}$ is the identity matrix, $A \in \mathbb{C^{L\times K}}, K>L$, $P \in \mathbb{C^{K\times K}}$, $\alpha$ is known and is fixed to a value and $\|.\|_F$ is the Frobenius norm. For example, if $P$ is a projection that projects onto $N(A)$, $AP=0$ which satisfies the objective function, and $\alpha = L$ in this case. However, my objective is not to force $AP$ to zero, hoping that $\alpha$ is less than $L$, also $\alpha$ does not have to be an integer and $P$ does not have to be a projection. Is this solvable? and if so, any directions on how to solve it would be much appreciated.

Answer 1

Over the real numbers:

Ok, assume that $\alpha<L$ and the matrices are real. Let $f(P)=tr(P^TA^TAP), g(P)=K-2tr(P)+tr(PP^T)-\alpha$. By the Lagrange's method, we seek $\lambda\in\mathbb{R}$ s.t. $Df_P+\lambda Dg_P=0$, that is, for any matrix $H$,

$tr(H^TA^TAP+P^TA^TAH)+\lambda(-2tr(H)+tr(PH^T+HP^T))=0$, that is

$tr(P^TA^TAH)=\lambda tr(H-P^TH)$. This is equivalent to $P^TA^TA=\lambda(I-P^T).$

Case 1. $\lambda=0$. Then $P^TA^TA=0$ and $f(P)=0$. It remains to find $P$ s.t.one has also $g(P)=\alpha$.

Case 2. $\lambda\not=0$. Then $P^T=(I+\dfrac{1}{\lambda} A^TA)^{-1}$ and $P$ is symmetric. It remains to find $\lambda\not= 0$ s.t. $(I+\dfrac{1}{\lambda} A^TA)$ is invertible and $K-2tr(P)+tr(P^2)=\alpha$.

EDIT: Now we are over $\mathbb{C}$. Remark 1. The case 1 is impossible. Proof: $A^HA$ is hermit $\geq 0$. Then we may assume $A^HA=diag(t_1,\cdots,t_L,0_{K-L})$ with $t_i>0$. Let $P$ s.t. $A^HAP=0$. Then the $L$ first rows of $P$ are $0$. Then $||I-P||^2\geq L>\alpha$.

Remark 2. For the case 2., we have a good candidate $P=(I+\dfrac{1}{\lambda} A^HA)^{-1}$. It can happen that there is a better complex solution, but it is unlikely (except if you are unlucky!). Here $P=diag(\dfrac{\lambda}{\lambda+t_1},\cdots,\dfrac{\lambda}{\lambda+t_L},I_{K-L})$ and $\lambda$ is a solution of $L+\sum_{i=1}^L \dfrac{\lambda^2}{(\lambda+t_i)^2}-2\sum_{i=1}^L \dfrac{\lambda}{\lambda+t_i}=\alpha$.