Let $G={\rm SU}(d)$ be the unitary group and $\rho(g)$ an irreducible representation of $g\in G$ in a $D$ dimensional Hilbert space $V$. Let $e_i\in V$ be the diagonal matrix whose only non-zero diagonal entry is the $i$-th element and $i=1,\dots,D$.

Suppose that I have $N$ discrete group elements $g_\alpha$ and $$E_{i\alpha} = \rho(g_\alpha)\,e_i\, \rho(g_\alpha)^\dagger.$$ What is the number of linearly independent operators? Or, alternatively, what is the number or orthogonal operators? Namely the number of linearly independent operators $A\in V$ such that $$ {\rm Tr}[A E_{i\alpha}] = 0, ~ ~ \forall i=1,\dots,D, \forall \alpha=1,\dots,N. $$

I have done numerical simulations with Haar randomly generated $g_\alpha$, and found that the number of orthogonal operators is always a constant number, for fixed $D,d,N$. Is there a way to calculate this number?

I expected that the minimal $N$ to obtain $D^2$ linearly independent operators $E_{i\alpha}$ was $N=D$, but that's not the case. For example for SU(2), $D=2J+1$ where $J$ is the spin and the minimal $N$ is $N=4J+1$. What is the reason behind this number?