$$\displaystyle\min_{\mathbf{P}} \mathbf{\|AP\|}_F^2 \quad{} \text{subject to} \quad{} \text{trace}(\mathbf{(I-P)(I-P)^H})=\alpha, \alpha \geq0$$ Can also be rewritten as $$\displaystyle\min_{\mathbf{P}} \mathbf{\|AP\|}_F^2 \quad{} \text{subject to} \quad{} \mathbf{\|I-P\|}_F^2=\alpha, \alpha \geq0$$ where $\mathbf{I}$ is the identity matrix, $A$ is $L\times K, K>L$, $P$ is $K \times K$, $\alpha$ is known and is fixed to a value and $\|.\|_F$ is the Frobenius norm. For example, if $P$ is a projection that projects onto $N(A)$, $AP=0$ which satisfies the objective function, and $\alpha = L$ in this case. However, my objective is not to force $AP$ to zero, hoping that $\alpha$ is less than $L$, also $\alpha$ does not have to be an integer and $P$ does not have to be a projection. Is this solvable? and if so, any directions on how to solve it would be much appreciated.