Here is an argument based on convex optimization.
Consider the case where $B$ has a one on the diagonal and the rest of the matrix is arbitrary. Without loss of generality, we can assume that $b_{11}=1$. Thus, we can write the general matrix $B=E+C$, where $E=e_1e_1^T$ with $e_1=(1,0,\ldots,0)$ being the first canonical basis vector, and $C$ is a symmetric matrix with $C_{11}=0$.
(Note: If the entry $\ge 1$ is an off-diagonal, then we can write $B=e_ie_j^T+e_je_i^T$, and run an argument similar to the one below---this one is somewhat more tedious so I did not work it out).
To save on typing, first define the notation:
$\DeclareMathOperator{\vect}{vec}$
\begin{equation*}
Z = A^{-1},\quad M = Z \otimes Z,\quad e=\vect(E)\quad c=\vect(C).
\end{equation*}
Then, the trace under question is:
\begin{equation*}
\mbox{trace}(Z(E+C)Z(E+C)) = e^TMe + c^TMc + 2c^TMe.
\end{equation*}
Actually, we have $(Kc)^TMc$ as the second term, where $K$ is the commutation matrix, but after some simplifications, it will turn out that we can drop $K$.
Now, we minimize $c^TMc + 2c^TMe$ subject to $c^Te=0$ (and the constraint that $Kc=c$ to ensure symmetry, but this constraint can be eliminated, so I've dropped it).
Introduce the Lagrange multiplier $\nu$ corresponding to the constraint $c^Te=0$, the
first-order optimality conditions for this convex optimization problem are:
\begin{equation*}
2Mc + 2Me - \nu e = 0,\qquad c^Te=0.
\end{equation*}
Thus, we have
\begin{eqnarray*}
Mc &=& (\nu/2) e - Me\\\\
c &=& (\nu/2) M^{-1}e - e\\\\
&\implies& c^Te = (\nu/2) e^TM^{-1}e - e^Te\\\\
&\implies& \nu=2\qquad\text{since}\ e^TM^{-1}e=1.
\end{eqnarray*}
Thus, in particular, an optimum $c$ must satisfy
\begin{equation*}
c = M^{-1}e - e,
\end{equation*}
from which it follows that $e^TMc = 1 - e^TMe$ and $c^TMc=0-c^TMe$. Thus, the objective is
\begin{equation*}
c^TMc + 2c^TMe= c^TMe = 1 - e^TMe,
\end{equation*}
from it immediately follows that
\begin{equation*}
\mbox{trace}(Z(E+C)Z(E+C)) = e^TMe + c^TMc + 2c^TMe \ge 1.
\end{equation*}
