Let $A$, $B$ be two $n\times n$ real matrices.

Let $A$ be a zero line-sum matrix where each row sum and each column sum equals zero, i.e., $$\sum_{i=1}^{n}a_{ij}=\sum_{j=1}^{n}a_{ij}=0 $$ (it seems there is no name for such matrix, see Name for matrices with vanishing row and column sums).

Let $B$ be a matrix where every entry is non-negative and each row sum and each column sum equals 1, i.e., $$\sum_{i=1}^{n}b_{ij}=\sum_{j=1}^{n}b_{ij}=1, b_{ij}\geq 0 $$ ($B$ is a doubly stochastic matrix).

An index set $S\subset[n]\times[n]$ is a permutation set if $S$ has $n$ elements and if for every $(i,j),(i'j')\in S$, $(i,j)\neq(i',j')$ then $i\neq i;$ and $j\neq j'$.

Now suppose there exists a permutation set $S$ such that $\sum_{(i,j)\in S} a_{ij}\geq \sum_{(i,j)\in S'} |a_{ij}|$ for every permuation set $S'$, and $\sum_{(i,j)\in S} b_{ij}\geq \sum_{(i,j)\in S'} b_{ij}$ for every permuation set $S'$. Is it true that $\langle A,B\rangle_F\geq 0$?

Note this is obviously true if $B$ is a permutation matrix or if $b_{ij}=1/n$ for every $(i,j)$. These seems like the two extreme cases. But how to prove it in general?