I'm trying to use the log-determinant to regularize an optimization problem. To make the argument work, I need to bound the second derivative of the log-determinant.

I need to prove that $\text{Tr}\left( \left(A^{-1} B\right)^2\right) \geq 1$ whenever:

1. $A$ is positive semidefinite
2. $B$ is symmetric, has zeroes on the diagonal, and has at least one entry which is $\geq 1$.
3. The diagonal entries of $A$ are 1.

From messing around a bit it seems like in fact there are always unit vectors $v, w$ such that $v^T A^{-1} B w \geq 1$---for example, this is true if B has only a single large entry, or if A and B commute. That would imply the conclusion and seems like it should be easy enough to confirm or deny, but I'm stumped. 

Any thoughts?