In the paper 

On the Order of Finitely Generated Subgroups of $\mathbb{Q}^{*}$ (mod $p$) and Divisors of $p-1$,  Journal of Number Theory 57, pp. 207-222

by F. Pappalardi, 

Let $a>1$ non-square. Denote by $l_a(p)$ the multiplicative order of $a$ modulo $p$. Then we have for some positive $\gamma$, 
    $$
    \sum_{p<x} \frac{1}{l_a(p)} \ll \frac{\sqrt{x}}{(\log x)^{1+\gamma}}.$$

It seems that breaking $\sqrt x$ on the upper bound is very difficult. 

If your assertion is true, then the upper bound would have $N \log\log x$ which is way beyond the upper bound $\sqrt x / (\log x)^{1+\gamma}$.