Evidence for integer factorization is in $P$

Peter Sarnak believes that integer factorization is in $P$. It is a well-known open problem in TCS to identify the real complexity class of integer factorization. Take a look at this link for Peter Sarnak's lectures where he mentions that he does not believe factoring is not in $P$.

What evidence is there that integer factorization is in $P$ other than the fact that polynomial factorization is in $P$?

-
The fact that primality is in $P$, and intuitively that feels like a rather big part of the problem? –  Federico Poloni Oct 28 '11 at 9:15
–  Kaveh Oct 28 '11 at 10:52