In the paper
On the Order of Finitely Generated Subgroups of $\mathbb{Q}^{*}$ (mod $p$) and Divisors of $p-1$, Journal of Number Theory 57, pp. 207-222
by F. Pappalardi,
Let $a>1$ non-square. Denote by $l_a(p)$ the multiplicative order of $a$ modulo $p$. Then we have for some positive $\gamma$, $$ \sum_{p<x} \frac{1}{l_a(p)} \ll \frac{\sqrt{x}}{(\log x)^{1+\gamma}}.$$
It seems that breaking $\sqrt x$ on the upper bound is very difficult.
If your assertion is true, then the upper bound would have $N \log\log x$ which is way beyond the upper bound $\sqrt x / (\log x)^{1+\gamma}$.