Your trace equals ${\rm tr} ((BPA^\top+APB^\top)X)$. This equals 0 for all symmetric matrices $X$ is and only if $C=BPA^\top+APB^\top=0$ (else take $X=C$, note that $C$ is symmetric). Of course, this is possible. For example, if $m=n$ and $P=\rm I$ is identity matrix, the product $BA^{\top}$ may be antisymmetric without problems.
Fedor Petrov
- 108.9k
- 9
- 264
- 459