Let $n:=N$. Let us show that for all natural $n$ and all $p\in(0,1)$
$$P(A_n>\sqrt p)\le\frac{\sqrt p}{1-\sqrt p+p},\tag{1}$$
so that $P(A_n>\sqrt p)\to0$ whenever $p\downarrow0$.

By [Cantelli's inequality][1],  
$$\begin{aligned}
P(A_n>\sqrt p)&\le\frac{p(1-p)/n}{p(1-p)/n+(\sqrt p-p)^2} \\ 
&\le\frac{p/n}{p/n+(\sqrt p-p)^2} \\ 
&=\frac{1/n}{1/n+(1-\sqrt p)^2} \\ 
&\le\frac{\sqrt p}{\sqrt p+(1-\sqrt p)^2} 
=\frac{\sqrt p}{1-\sqrt p+p}
\end{aligned}
$$
if $n\sqrt p\ge1$, 
in which case (1) holds.  

In the remaining case, when $n\sqrt p<1$, we have 
$$P(A_n>\sqrt p)=P(A_n>0)=1-(1-p)^n\le np<\sqrt p<\frac{\sqrt p}{1-\sqrt p+p},$$
so that (1) again holds.  

Thus indeed, (1) holds for all natural $n$ and all $p\in(0,1)$. (It actually holds for all $p\in[0,1]$.)

[1]: https://en.wikipedia.org/wiki/Cantelli%27s_inequality