Is there any evidence for or against the idea that some of the important statements of number theory that have only been proved using infinite sets, are in fact undecidable in Peano arithmetic?

Most modern number theory is based, not on considering problems directly, but on applying some sort of complicated machinery. In particular, one works with infinite sets with various sorts of structure, like algebraic number fields, modular forms, and the Riemann zeta function.

I have sometimes wondered whether, with sufficient work and cleverness, one could reconstruct these proofs entirely in the language of elementary number theory. Theoretically, this should be possible, except for the fact that set theory is a stronger theory than Peano arithmetic or something else that represents elementary number theory. An example of this is the proof of the Ramsey theorem by way of the infinite Ramsey theorem. This proof cannot really be translated into arithmetic since it can also be used to prove the strengthened finite Ramsey theorem, which is undecidable in PA.

It is natural to conjecture that other results, like results about the distribution of primes which depend on zeta functions and related functions may also be undecidable. Is there any hard evidence that this is or is not the case?