Is this estimate true? Can anyone give a proof of it?
$$
\sum_{a=1}^{p-1}\sum_{b=1}^{p-1}\frac{b}{a(ab)_p}=\frac{1}{2}p\ln^2 p+o(p\ln^2 p)\qquad (p\text{ prime, } p\to\infty)
$$
where $
(ab)_p\equiv ab\;(\operatorname{mod}p)$, $0<(ab)_p<p$.
Is this estimate true? Can anyone give a proof of it?
$$
\sum_{a=1}^{p-1}\sum_{b=1}^{p-1}\frac{b}{a(ab)_p}=\frac{1}{2}p\ln^2 p+o(p\ln^2 p)\qquad (p\text{ prime, } p\to\infty)
$$
where $
(ab)_p\equiv ab\;(\operatorname{mod}p)$, $0<(ab)_p<p$.
Not an answer but a simple heuristic argument: if you set $r=(ab)_p$, the OP's sum is equal to $$\sum_{1\le r\le p-1}\dfrac{1}{r}\sum_{1\le a\le p-1}\dfrac{1}{a}(ra^{-1})_p$$ This proves immediately that the sum is less than $(p-1)H_{p-1}^2$, asymptotically $p\log(p)^2$, and if we assume (heuristic part) that $(ra^{-1})_p$ has average $(p-1)/2$ we indeed obtain a guess of $p\log(p)^2/2$. Maybe this last part can be made rigorous.
EDIT: if you consider the much simpler SINGLE sum $S(p)=\sum_{1\le a\le p-1}\dfrac{(a^{-1})_p}{a}$, the same heuristic would give an asymptotic of $p\log(p)/2$. However, numerically $S(p)/(p\log(p))$ does NOT seem to tend to a limit, but oscillates between something like $0.38$ and $0.52$. This should be much easier to analyze, and perhaps indicate that there is also some oscillation in the OP's original question, with no limit.
Not an answer, but an argument that your sum is between $(1/4+o(1))p\log^2(p)$ and $(3/4+o(1))p\log^2(p)$.
Write $S(p)$ for your sum. Separate the sum into pieces according to the integer part of $(ab/p)$: $$ \begin{align*} S(p)&=\sum_{a=1}^{p-1}\frac{1}{a}\sum_{k=0}^{a-1} \sum_{kp/a<b<(k+1)p/a} \frac{b}{ab-kp}\\ &\sim\sum_{a=1}^{p-1}\frac{1}{a}\sum_{k=0}^{a-1} \frac{kp}{a}\sum_{kp/a<b<(k+1)p/a} \frac{1}{ab-kp}.\\ \end{align*} $$ The innermost sum is the sum of reciprocals of integers in an arithmetic progression. The first term in the progression is $(-pk)_a$, and the sum of the reciprocals of the other terms in the progression is $\log(p/a)/a+O(1/a)$, so $$ \begin{align*} S(p)&\sim\sum_{a=1}^{p-1}\frac{1}{a}\sum_{k=0}^{a-1} \frac{kp}{a}\left[\frac{1}{(-pk)_a}+\frac{1}{a}\log\left(\frac{p}{a}\right)\right]\\ &=p\sum_{a=1}^{p-1}\frac{1}{a^3}\log(p/a)\sum_{k=0}^{p-1}k+p\sum_{a=1}^{p-1}\frac{1}{a^2}\sum_{k=1}^{a-1}\frac{k}{(-pk)_a}\\ &=\frac{1}{4}p\log^2(p)+p\sum_{a=1}^{p-1}\frac{1}{a^2}\sum_{k=1}^{a-1}\frac{k}{(-pk)_a}. \end{align*} $$ This gives the claimed lower bound for $S(p)$. For the upper bound, we observe that for fixed $a$, we have $\{(-pk)_a:1\leq k\leq a-1\}= \{1,\ldots,a-1\}$. So an upper bound for the second term above is $$ p\sum_{a=1}^{p-1}\frac{1}{a^2}\sum_{k=1}^{a-1}\frac{k}{a-k}\sim \frac{1}{2}p\log^2(p). $$
Not really an answer, but here is the plot for the first 200 primes:
Maple code:
with(plots):
f := proc(p)
option remember;
return evalf(add(add(b/a/modp(a*b,p),b=1..p-1),a=1..p-1)/p/ln(p)^2);
end:
listplot([seq(f(ithprime(k)),k=10..200)],style=point);
(Obviously this is completely unintelligent, and much more efficient methods are doubtless possible.)
I feel obliged to flesh out my comments (and to modify my wrong answer, thanks to GH from MO).
Write $n=ab$, and let $H_{k}$ the $k$-th harmonic number, $\tau(n)$ the number of positive divisors of $n$. LHS is greater than the related sum $\sum_{a=1}^{p-1}\sum_{b=1}^{a}\dfrac{b}{a(ab)_{p}}$, that we'll denote by $S$.
Provided the limit as $\tau(n)$ tends to $\infty$ of $f(n):=\frac{2}{\tau(n)}\sum_{d\mid n,d\leqslant\sqrt{n}}\dfrac{d^{2}}{n}$ exists and equals a positive constant $M$, we have:
$S=\sum_{n=1}^{(p-1)^2}\sum_{d\mid n,d\leqslant\sqrt{n}}\dfrac{d^2}{n(n)_{p}}\sim\sum_{n=1}^{(p-1)^2}\sum_{d\mid n,d\leqslant\sqrt{n}}\dfrac{M}{(n)_{p}}\sim\sum_{n=1}^{(p-1)^2}\sum_{d\mid n,d\leqslant\sqrt{n}}M\dfrac{H_{p-1}}{p-1}\sim M\sum_{n=1}^{(p-1)^2}\sum_{d\mid n,d\leqslant\sqrt{n}}\dfrac{\log(p-1)}{p-1}$.
The idea is to consider all divisors of $n$ less than its square root, and to replace $\dfrac{1}{n_p}$ by its average value, which is $\dfrac{H_{p-1}}{p-1}\sim\dfrac{\log(p-1)}{p-1}$ (in my comment I erroneously took the reciprocal of the average value and not the average value of the reciprocals, hence the missing $\log(p-1)$).
We thus obtain $S\sim M\dfrac{\log(p-1)}{p-1}\sum_{n=1}^{(p-1)^2}\dfrac{\tau(n)}{2}$. As $D(x):=\sum_{n=1}^{x}\tau(n)$ is provably asymptotic to $x(\log x+K)$ where $K$ is a positive constant (see Dirichlet divisor problem on Wikipedia), we end up with:
$S\sim\frac{M}{2}\frac{\log(p-1)}{p-1}(p-1)^2(\log (p-1)^2+K)\sim Mp\log^{2} p$ which provides a lower bound for the sum of the OP of the desired order of magnitude.
Edit: the following link: http://www.les-mathematiques.net/phorum/read.php?5,1967230,1967504#msg-1967504 shows that if $M$ exists, it equals $\frac{1}{2}$.