Another modest suggestion: Set $f(n)=\sum_{i=1}^{n-1}\lfloor i^2/n\rfloor-(n-1)(n-2)/3$. Then it seems to be the case that $f(ab)\ge f(a)+f(b)$ for all positive integers $a$ and $b$. (Checked for all $a,b$ where $ab\le75000$.) Since $f(n)\ge0$ for all primes $n$, one is reduced to prove this inequality. Maybe some clever manipulation of \begin{equation} \sum_{i=1}^{ab-1}\left\lfloor\frac{i^2}{ab}\right\rfloor=\sum_{0\le i<b, 0\le j<a}\left\lfloor\frac{(ai+j)^2}{ab}\right\rfloor \end{equation} yields some progress/insight?
Two more empirical observations (similar to Henri Cohen's comment regarding $p^n$ for $p\equiv3\pmod{4}$): It seems to be that \begin{equation} f(3^n) = \frac{5 - (-1)^n}{2}\cdot3^{\frac{2n - 1 + (-1)^n}{4}} - 2 \end{equation}\begin{equation} f(3^n) = \frac{5 - (-1)^n}{6}\cdot3^{\lfloor n/2\rfloor} - \frac{2}{3} \end{equation} and \begin{equation} f(p^n) = 3\cdot\frac{p^{\lfloor n/2\rfloor}-1}{2} \end{equation}\begin{equation} f(p^n) = \frac{p^{\lfloor n/2\rfloor}-1}{2} \end{equation} for $p\equiv1\pmod 4$.