Fedor's proof can be generalized :

Let $S = X^{1/2}$, $U_i = V_i S^{1/2}$ and $W_i = U_i (U_i^T S U_i)^{-1/2}$ .
Then $W_i^T S W_i = I_m$ where $I_m$ is the $m\times m$ identity .

Since $log(x) \ge 1 - 1/x$ it is enough to show that 
$$tr \sum_{i=1}^N \mu_i \le 1$$ 
where 
$$\mu_i = (\sum_{j=1}^N W_i^T W_j W_j^T W_i)^{-1}$$ .

Then 
$$tr \sum_{i=1}^N \mu_i = tr \sum_{i=1}^N \mu_i^2 \mu_i^{-1}$$
$$= tr [\sum_{i=1}^N \mu_i^2 \sum_{j=1}^N W_i^T W_j W_j^T W_i]$$
$$\ge tr \sum_{i=1}^N \sum_{j=1}^N W_i \mu_i W_i^T W_j \mu_j W_j^T$$
$$= tr [(\sum_{i=1}^N W_i \mu_i W_i^T)^2]$$
$$\ge [tr (S \sum_{i=1}^N W_i \mu_i W_i^T)]^2$$
$$= (tr \sum_{i=1}^N \mu_i)^2$$ ,

where I have used that
$$tr (\mu_i^2 W_i^T W_j W_j^T W_i) + tr (\mu_j^2 W_j^T W_i W_i^T W_j) \ge 2 tr (W_i \mu_i W_i^T W_j \mu_j W_j^T)$$ .

This follows from
$$0 \le tr[(W_i^T W_j \mu_j - \mu_i W_i^T W_j) (\mu_j W_j^T W_i - W_j^T W_i \mu_i)]$$ .