Numerical results give random picture. If it is really the case then usual heuristic arguments confirm your conjecture.

ListPlot[Table[FractionalPart[Sqrt[PartitionsP[i]]], {i, 1, 5000}]]

[![enter image description here][1]][1]

Distance $0$ number is 1: $p(1)=1^2$. Distance 1 numbers:
\begin{gather}p(2)=1^2+1,\quad p(3)=2^2-1,\quad p(4)=2^2+1,\\p(7)=4^2-1,\quad p(13)=10^2+1,\quad p(35)=122^2-1.\end{gather}
Distance 2 numbers:
$$p(5)=3^2-2,\quad p(6)=3^2+2,\quad p(20)=25^2+2.$$


  [1]: https://i.sstatic.net/qw3b8.jpg