Let $n:=N$. Let us show that for all natural $n$ and all $p\in(0,1)$
$$P(A_n>\sqrt p)\le\frac{\sqrt p+p}{1+p},\tag{1}$$
so that $P(A_n>\sqrt p)\to0$ whenever $p\downarrow0$.

By [Cantelli's inequality][1], if $n\sqrt p\ge1$, then
$$\begin{aligned}
P(A_n>\sqrt p)&\le\frac{p(1-p)/n}{p(1-p)/n+(\sqrt p-p)^2} \\ 
&\le\frac{p(1-p)\sqrt p}{p(1-p)\sqrt p+(\sqrt p-p)^2} \\  
&=\frac{\sqrt p+p}{1+p}, 
\end{aligned}
$$
so that (1) holds if $n\sqrt p\ge1$.  

In the remaining case, when $n\sqrt p<1$, we have 
$$P(A_n>\sqrt p)=P(A_n>0)=1-(1-p)^n\le np<\sqrt p<\frac{\sqrt p+p}{1+p},$$
so that (1) again holds.  

Thus indeed, (1) holds for all natural $n$ and all $p\in(0,1)$. (It actually holds for all $p\in[0,1]$.)

[1]: https://en.wikipedia.org/wiki/Cantelli%27s_inequality