Let $n:=N$. Let us show that for all natural $n$ and all $p\in(0,1)$
$$P(A_n>\sqrt p)\le\frac{\sqrt p}{(1-\sqrt p)^2},\tag{1}$$
so that $P(A_n>\sqrt p)\to0$ whenever $p\downarrow0$.

By Chebyshev's inequality, 
$$P(A_n>\sqrt p)\le\frac{p(1-p)/n}{(\sqrt p-p)^2}\le\frac{p/n}{(\sqrt p-p)^2}
=\frac{1/n}{(1-\sqrt p)^2}\le\frac{\sqrt p}{(1-\sqrt p)^2}$$
if $n\sqrt p\ge1$, in which case (1) holds.  

On the other hand, if $n\sqrt p<1$, then 
$$P(A_n>\sqrt p)=P(A_n>0)=1-(1-p)^n\le np<\sqrt p,$$
so that (1) again holds.  

Thus indeed, (1) holds for all natural $n$ and all $p\in(0,1)$.