Let $n:=N$. Let us show that 
$$P(A_n>\sqrt p)\to0$$ whenever $p\downarrow0$.
By Hoeffding's bound [(2.13)][1] (published as [(2.9)][2]) with $b=1$, $t=\sqrt p-p\sim\sqrt p$, and $\sigma^2=p(1-p)\sim p$, we have 
$$P(A_n>\sqrt p)\le\exp\{-(1+o(1))n\sqrt p\,\ln\frac1{\sqrt p}\}.$$
So, $P(A_n>\sqrt p)\to0$ if $n\sqrt p\ge1$. 

On the other hand, if $n\sqrt p<1$, then $P(A_n>\sqrt p)=P(A_n>0)=1-(1-p)^n\le np\to0$. 

Thus indeed,
$P(A_n>\sqrt p)\to0$ whenever $p\downarrow0$. 

[1]: https://repository.lib.ncsu.edu/bitstream/handle/1840.4/2170/ISMS_1962_326.pdf;jsessionid=E826CD4073F94D7C93CC8C8B75C3B814?sequence=1

[2]: https://www.jstor.org/stable/2282952?seq=1