This is a question I also asked on MSE . If it is frowned upon to ask the same question on both threads, you can vote to close this thread.
Let $n \in \mathbb {N^*}$ and $$S_n = \sum_{k=1}^{n-1} (k^2 \bmod n)$$
with the first values 0, 1, 2, 2, 10, 13, 14, 12, 24, 45, 44, 38, 78, 77, 70, 56, 136, 129, 152, 130, 182, 209, 184, 148, 250, 325, 288, 294, 406, 365, 372, 304, 484, 561, 490, 402, 666, 665, 572, 540, 820, 805, 860, 726, 840, 897, 846, 680, 980, 1125
and it appears that $$S_n\leq \frac {n(n-1)}2$$
We can show that $$S_n\leq \frac {n(n-1)}2\iff \sum _{i=1}^{n-1}\left\lfloor \frac{i^2}{n}\right\rfloor \ge \frac{\left(n-1\right)\left(n-2\right)}3$$ we have equality if $n$ is prime and $n \equiv 1 \mod 4$
Do you think this inequality is true?
Addition :
We fix a prime number $p$ that satisfies $p \equiv 1 \pmod{4}$. Let $\forall n \in \mathbb{N}^*$, $S_n$ be defined as the sum from $k = 0$ to $p^n - 1$ of $(k^2)_{p^n}$. Numerically, it appears that $2S_n = p^n(p^n - p^{\lfloor n/2 \rfloor})$. If we can prove this equality, we can deduce inequality (the object of the thread) in many cases
Addition 2 For n odd, we can show that $$S_n = \frac{n(n-1)}{2} + \sum_{\substack{d \mid n \\ d > 1}} \mu(d)^2 \sum_{k=1}^{n-1} k \left( \frac{k}{d} \right) $$, but how to prove that $\displaystyle \sum_{\substack{d \mid n \\ d > 1}} \mu(d)^2 \sum_{k=1}^{n-1} k \left( \frac{k}{d} \right)\leq 0$