A statement referring to an infinite set can sometimes be logically rephrased using only finite sets/objects. For example, "The set of primes is infinite" <-> "There is no largest prime". Pleasantly, the proof of this statement does not seem to need infinity either (assume a largest prime, contradiction).

What reason is there, other than convenience or curiosity, to adjoin infinite sets to our universe by axiomatically declaring that one exists?

Specifically:

What is an example of a theorem in ZF or ZFC which 1) Does not refer to infinite sets, but 2) Cannot be proven if the Axiom of Infinity is excluded?

(See Zermelo–Fraenkel set theory for the Axiom of Infinity in context.)