In this paper of mine (not yet published) : http://www.math.ucla.edu/~i707107/SJK_TmImprovement.pdf
I prove that
Let $a>1$. Denote by $l_a(p)$ the multiplicative order of $a$ modulo $p$. Then we have $$ \sum_{p<x} \frac{1}{l_a(p)} \leq \left(\frac{2}{\pi}\sqrt{6\log a} +o(1)\right)\frac{\sqrt{x}}{\log x}.$$
using Z. Engberg's upper bound
$$\sum_{l_a(p)=d} 1 \leq \frac{ \varphi(d)\log a}{2\log d} + O\left(\frac{d\log\log d}{(\log d)^2}\right).$$
It seems that breaking $\sqrt x$ on the upper bound is very difficult.
If your assertion is true, then the upper bound would have $N \log\log x$ which is way beyond the upper bound of mine.