I hope, that below is the proof in general case, but I am not sure, so leave another answer for the case $N=2$ still.
Denote $X^{1/4} v_i=u_i$, $X^{1/2}=S$, then we have ${\rm tr}\,S^2=1$ and need to prove that $$ \prod_i \sum_j \frac{(u_i,u_j)^2}{(Su_i,u_i) (Su_i,u_i)}\geqslant \frac{N^N}{({\rm tr}\,S^2)^N}. $$ By homogeneity in $u$'s we may suppose that $(Su_i,u_i)=1$ for all $i$. On the set of symmetric operators on $\mathbb{R}^n$ we have an inner product $(Y,Z)={\rm tr}\, YZ$, so ${\rm tr}\,S^2=\|S\|^2$. Our restriction on $S$ may be rewritten as $(S,u_i\otimes u_i)=1$, where we naturally identify $u\otimes u$ with the (rank at most 1) operator $x\rightarrow (u,x)u$. For operators $T_i=u_i\otimes u_i$, $i=1,\dots,N$ we have $(u_i,u_j)^2=(T_i,T_j)^2$. That is, we get the following problem: given that $(T_i,T_j)\geqslant 0$ for all $i,j$ and $(S,T_i)=1$ prove that $$\|S\|^{2N}\prod_{i=1}^N (T_i,T_1+\dots+T_N)\geqslant N^N.$$ We use quasilinearization techniques. Namely, choose numbers $\mu_1,\dots,\mu_n$ such that $\prod \mu_i=1$ and all numbers $\mu_i^2 (T_i,T_1+\dots+T_N)$ are equal. This is possible since all these numbers are positive (they are sums squares including strictly positive $(T_i,T_i)$). Then geometric mean of these numbers equals their arithmetic mean and we have $$ \|S\|^2\left(\prod_{i=1}^N (T_i,T_1+\dots+T_N)\right)^{1/N}=\|S\|^2\frac{\sum \mu_i^2 (T_i,T_1+\dots+T_N)}{N}, $$ and our aim is to prove $$\|S\|^2 \sum_i\sum_j \mu_i^2(T_i,T_j)\geqslant N^2.$$ Using $\mu_i^2+\mu_j^2\geqslant 2\mu_i\mu_j$ we see that it suffices to prove $$\|S\|^2\|\sum \mu_i T_i\|^2\geqslant N^2.$$ Product of norms of $S$ and $\sum \mu_i T_i$ is not less than their inner product, which equals $\sum \mu_i$, which is at least $N$, since $\prod \mu_i=1$.