There's a remarkably simple formula that explains this behavior. If $x,y,x_i$ are drawn IID from a Gaussian distribution in $\mathbb{R}^d$, we have the following:
$$\mathbb{E}\left\|\sum_i^b x_i x_i^T\right\|_F^2=b\ \mathbb{E}(\|x\|^4)+b(b-1) \mathbb{E}\langle x, y\rangle$$
This means that the value of $\frac{1}{b^2}\|XX^T\|_F^2$ is a weighted mean between $E[\|x\|^4]$ and $\mathbb{E}\langle x, y\rangle$.
Sometimes we can exchange the order of $E$ and reciprocal, which would make quantity in question a weighted harmonic mean between $1/E[\|x\|^4]$ and $1/\mathbb{E}\langle x, y\rangle$, which is the empirical behavior observed in plots above.
Proof:
Matrix multiplication is commutative w.r.t to any Schatten norm, hence we can write $$\left\|\sum_i^b x_i x_i^T\right\|=\left\|X^TX\right\|^2_F=\left\|XX^T\right\|^2_F=\sum_i \sum_j \langle x_i, x_j\rangle^2$$
Here $X$ is formed by stacking $x_i$ as rows, as per-convention for the data matrix in statistics.
Taking the expectation of latter expression we have $b^2$ terms. Because $x_i$ are sampled IID, there are only two kinds of terms: $b$ diagonal terms and $b(b-1)$ off-diagonal terms, result follows.
Alternative ways of writing this result using result for dot produt
$$\mathbb{E}\left\|\sum_i^b x_i x_i^T\right\|_F^2=b\ \mathbb{E}(\|x\|^4)+\frac{1}{2}b(b-1)\left(\mathbb{E}\left[\|x\|^4\right]-\mathbb{E}\left[\|x\|^2\right]^2+2\ \|\mathbb{E}x\|^4\right) $$
Or it can be written in terms of covariance matrix/mean using Gaussian moment result here