The proof of the general case, in a strong form suggested in the end of OP.

Denote $X^{1/4} v_i=u_i$, $X^{1/2}=S$, then we have ${\rm tr}\,S^2=1$ and need to prove that 
$$
{\rm tr}\,S^2\geqslant 
\sum_i \frac{(Su_i,u_i) (Su_j,u_j)}{\sum_j (u_i,u_j)^2},
$$
then the very original inequality follows by applying AM-GM to $N$ summands in RHS.

By homogeneity in $u$'s we may suppose that $(Su_i,u_i)=1$ for all $i$. On the set of symmetric operators on $\mathbb{R}^n$ we have an inner product $(Y,Z)={\rm tr}\, YZ$, so ${\rm tr}\,S^2=\|S\|^2$. Our restriction on $S$  may be rewritten as $(S,u_i\otimes u_i)=1$, where we naturally identify $u\otimes u$ with the (rank at most 1) operator $x\rightarrow (u,x)u$. For operators $T_i=u_i\otimes u_i$, $i=1,\dots,N$ we have $(u_i,u_j)^2=(T_i,T_j)$. That is, we get the following problem: given that $(T_i,T_j)\geqslant 0$ for all $i,j$ and $(S,T_i)=1$ prove that 
$$
\|S\|^2 \geqslant \sum_{i=1}^N (T_i,T_1+\dots+T_N)^{-1}
$$
For $c_i=(T_i,\sum T_j)$ we choose numbers $\mu_1,\dots,\mu_N$ such that $\sum 1/c_i=(\sum \mu_i)^2/\sum \mu_i^2c_i$, for example take $\mu_i=1/c_i$. Then 
$$
\|S\|^2\sum \mu_i^2c_i=\|S\|^2 \sum_i \sum_j \mu_i^2 (T_i,T_j)\geqslant \|S\|^2\|\sum \mu_iT_i\|^2\geqslant (S,\sum \mu_iT_i)^2=(\sum \mu_i)^2
$$
as desired (we have used that $\mu_i^2 (T_i,T_j)+\mu_j^2 (T_j,T_i)\geqslant 2\mu_i\mu_j (T_i,T_j)$ for all pairs $i\ne j$.)