Let $n:=N$. Let us show that for all natural $n$ and all $p\in(0,1)$ $$P(A_n>\sqrt p)\le\frac{\sqrt p+p}{1+p},\tag{1}$$ so that $P(A_n>\sqrt p)\to0$ whenever $p\downarrow0$.
Consider first the case when $n\ge1/\sqrt p$, so that $1/n\le\sqrt p$. In view of Cantelli's inequality, $$\begin{aligned} P(A_n>\sqrt p)&\le\frac{p(1-p)/n}{p(1-p)/n+(\sqrt p-p)^2} \\ &\le\frac{p(1-p)\sqrt p}{p(1-p)\sqrt p+(\sqrt p-p)^2} \\ &=\frac{\sqrt p+p}{1+p}, \end{aligned} $$ so that (1) holds if $n\ge1/\sqrt p$.
In the remaining case, when $n<1/\sqrt p$, we have $\sqrt p<1/n$ and hence $$P(A_n>\sqrt p)=P(A_n>0)=1-(1-p)^n\le np<\sqrt p<\frac{\sqrt p+p}{1+p},$$ so that (1) again holds.
Thus indeed, (1) holds for all natural $n$ and all $p\in(0,1)$. (It actually holds for all $p\in[0,1]$.)