This issue came up both in the logic course that I taught last semester and in the set theory course that I am teaching this semester.
What I told the students is that in order to do mathematical logic, we need a basic understanding of words over a finite alphabet. We cannot build a theory from less than that.
But if we understand finite strings, we basically have the natural numbers.
Some parts of mathematical logic assume some basic set theory, such as the completeness theorem of first order languages over uncountable alphabets (or just alphabets that are not recursively enumerable). But this can be avoided if you stick to sufficiently simple
alphabets (or even finite alphabets).
Similarly, you cannot do axiomatic set theory without a basic understanding of logic, which in turn requires a basic understanding of strings.
On the other hand, once you have built a sufficient theory of logic and set theory, you can use that in order to analyse mathematics. This is somewhat similar to the way that we learn mathematics: You learn to add natural numbers first, and then (usually something like 12 or more years after that) you learn about Peano Axioms that put everything on a solid foundation. I believe that this sort of circle cannot be avoided.