Numerical results give random picture. If it is really the case then usual heuristic arguments confirm your conjecture.
ListPlot[Table[FractionalPart[Sqrt[PartitionsP[i]]], {i, 1, 5000}]]
Distance $0$ number is 1: $p(1)=1^2$. Distance 1 numbers: \begin{gather}p(2)=1^2+1,\quad p(3)=2^2-1,\quad p(4)=2^2+1,\\p(7)=4^2-1,\quad p(13)=10^2+1,\quad p(35)=122^2-1.\end{gather} Distance 2 numbers: $$p(5)=3^2-2,\quad p(6)=3^2+2,\quad p(20)=25^2+2.$$