Does finite math need the Axiom of Infinity? "maybe you could do it without" - I don't think you can. Goodstein's Theorem cannot be proved in PA, and I'm quite sure it cannot be proved in ZFC minus the axiom of infinity either.

Examples of great mathematical writing Although my French is fairly weak, I was able to read and immensely enjoy Serre's "Cours d'Arithmetique". Clear, motivated, engaging and simply memorable. "Trees" is incredible.