Let $n:=N$. Let us show that for all natural $n$ and all $p\in(0,1)$ $$P(A_n>\sqrt p)\le\frac{\sqrt p}{1-\sqrt p+p},\tag{1}$$ so that $P(A_n>\sqrt p)\to0$ whenever $p\downarrow0$.
By Cantelli's inequality,
$$\begin{aligned}
P(A_n>\sqrt p)&\le\frac{p(1-p)/n}{p(1-p)/n+(\sqrt p-p)^2} \\
&\le\frac{p/n}{p/n+(\sqrt p-p)^2} \\
&=\frac{1/n}{1/n+(1-\sqrt p)^2} \\
&\le\frac{\sqrt p}{\sqrt p+(1-\sqrt p)^2}
=\frac{\sqrt p}{1-\sqrt p+p}
\end{aligned}
$$
if $n\sqrt p\ge1$,
in which case (1) holds.
In the remaining case, when $n\sqrt p<1$, we have $$P(A_n>\sqrt p)=P(A_n>0)=1-(1-p)^n\le np<\sqrt p<\frac{\sqrt p}{1-\sqrt p+p},$$ so that (1) again holds.
Thus indeed, (1) holds for all natural $n$ and all $p\in(0,1)$. (It actually holds for all $p\in[0,1]$.)