A "philosophy of math" tag would have been a good idea.
To answer your question, take a look at Hatry Field's "Science without Numbers". Also, Edward Nelson has developed a theory of "proto-integers", which is discussed in "Diffusion, Quantum Theory, and Radically Elementary Mathematics.Mathematics" (MN-47).
In my opinion, most attempts at nominalism (be it about abstract objects, infinite sets, or numbers) spend a lot of energy, and for what in the end? Moreover, the approach invariably comes across as a cheap trick (to me). This is my personal opinion, but I get the impression it is shared by many mathematicians.
I would quote Alonzo Church here, but the current climate does not allow for that anymore.