The Rabinowitsch criterion says that the discriminant $\Delta=-163$ is not a square modulo any prime $p<41$, see this nice survey by Pete Clark. So every prime factor of $P(x)$ must be $\geq 41$. But for the primes $p=43,47,53,\ldots$ it is easy to find the roots of $P(x)$ mod $p$, thanks to the fact that these primes are the first values of $P(x)$. For $p=43$ the roots are $\{1,-2\}$, for $p=47$ they are $\{2,-3\}$ and so on. This provides better bounds for the possible prime factors of $P(x)$ for $x>41$ and should explain your observation.

The Rabinowitsch criterion says that the discriminant $\Delta=-163$ is not a square modulo any odd prime $p<41$, see this nice article by Pete Clark. So every prime factor of $P(x)$ must be $\geq 41$. But for the primes $p=43,47,53,\ldots$ it is easy to find the roots of $P(x)$ mod $p$, thanks to the fact that these primes are the first values of $P(x)$. For $p=43$ the roots are $\{1,-2\}$, for $p=47$ they are $\{2,-3\}$ and so on. This provides better bounds for the possible prime factors of $P(x)$ for $x>41$ and should explain your observation.