Don't the axioms of set theory implicitly assume numbers? When one writes down the axioms of ZFC, or any other axiomatic theory for that matter, and making statements like "let x, y ..."  doesn't this assume an understanding (and thus existence) of natural numbers implicitly?  (Q1)
How is the reader to interpret statements such as existence of separate symbols, nevermind sets, without an intuitive notion of numbers?
Bourbaki talks about this within the framework of metamathematics, but then declares that the reader can read words, differentiate between different words etc. and that to assume otherwise is idiotic.
Is there an introduction to these circle of ideas & debates somewhere you'd recommend? (Q2)
 A: Note that there are different ways of thinking about set theory and more generally about Logic. They can be thought as a foundation for mathematics or they can be thought as a part of mathematics. If you are thinking of them as a foundation, at the end you have to accept some intuitive concepts, the point of foundation is not that it does not assume anything and builds on nothing, the point is that it is based on accepted theories. Almost all of mathematicians accept the very weak theories about natural numbers and they are sufficient for building the needed metamathematics for set theory. (Primitive Recursive Arithmetic would suffice but even weaker theories are sufficient).
I would suggest the introduction of K. Kunen's "Set Theory" book (the part he discusses the formalist viewpoint) and S.C. Kleene's "Metamathematics".
A: It seems that Deniz is raising a slightly uncomfortable question of whether some circularity is built into mathematical foundations. A similar common-sense circularity is what might be called the "paradox of the dictionary": since all words are defined in terms of other words, either dictionaries are hopelessly circular, or some words need to be left undefined in order to break out of the impasse. 
As it happens, I am preparing an article for eventual exportation to the nLab which at the outset deals with precisely this question. In the present draft, I have this passage: 

Logical foundations avoids this paradox ultimately by being concrete. We may put it this way: logic at the primary level consists of instructions for dealing with formal linguistic items, but the concrete actions for executing those instructions (electrons moving through logic gates, a person unconsciously making an inference in response to a situation) are not themselves linguistic items, not of the language. They are nevertheless as precise as one could wish. 
  $$ $$
  We emphasize this point because in our descriptions below, we obviously must use language to describe logic, and some of this language will look just like the formal mathematics that logic is supposed to be prior to. Nevertheless, the apparent circularity should be considered spurious: it is assumed that the programmer who reads an instruction such as "concatenate a list of lists into a master list" does not need to have mathematical language to formalize this, but will be able to translate this directly into actions performed in the real world. However, at the same time, the mathematically literate reader may like having a mathematical meta-layer in which to understand the instructions. The presence of this meta-level should not be a source of confusion, leading one to think we are pulling a circular "fast one". 

In other words, to break out of the circularity, it is enough to observe that computers can be programmed to recognize certain strings as well-formed terms or formulas (of a given axiomatic theory), and how to recognize inferences as valid. It's not as if there needs to be some background theory, or the prior existence of a completed or actual infinity of all expressions which might come up, sitting inside the computer. The computer is programmed to handle finite parts of the theory correctly, and the same applies to human users of a theory (although we say "taught", not "programmed"). 
A: This issue came up both in the logic course that I taught last semester and in the set theory course that I am teaching this semester.     
What I told the students is that in order to do mathematical logic, we need a basic understanding of words over a finite alphabet.  We cannot build a theory from less than that.
But if we understand finite strings, we basically have the natural numbers.
Some parts of mathematical logic assume some basic set theory, such as the completeness theorem of first order languages over uncountable alphabets (or just alphabets that are not recursively enumerable).  But this can be avoided if you stick to sufficiently simple
alphabets (or even finite alphabets).  
Similarly, you cannot do axiomatic set theory without a basic understanding of logic, which in turn requires a basic understanding of strings.  
On the other hand, once you have built a sufficient theory of logic and set theory, you can use that in order to analyse mathematics.  This is somewhat similar to the way that we learn mathematics:  You learn to add natural numbers first, and then (usually something like 12 or more years after that) you learn about Peano Axioms that put everything on a solid foundation.  I believe that this sort of circle cannot be avoided. 
A: If i may add (although this post is kind of old)
This circularity "problem" (if one wishes to see it as such), appears in (what is refered as) classical mathematics.
Now one should bear in  mind that even these classical mathematics (continuing along the lines of aristotle, euclid, archimedes, leibniz, cantor, hilbert, russel, goedel, etc..), have been cutoff and formalised (or sterilised if you like) to a greater extened that originaly meant.
In any case this is not the main argument.
But i would like to draw attention to the intuitionistic flavor of mathematics (and especially of the LEJ Brouwer path) (see for example LEJ Brouwer, Cambridge Lectures on Intuitionism, most of first lecture plus the appendix on marxists.org).
There Brouwer, aware of the problem, explicitly takes on the issue of mathematics over language or syntax.
Excerpt:

FIRST ACT OF INTUITIONISM
Completely separating mathematics from mathematical language and hence
  from the phenomena of language described by theoretical logic,
  recognising that intuitionistic mathematics is an essentially
  languageless activity of the mind having its origin in the perception
  of a move of time. This perception of a move of time may be described
  as the falling apart of a life moment into two distinct things, one of
  which gives way to the other, but is retained by memory. If the twoity
  thus born is divested of all quality, it passes into the empty form of
  the common substratum of all twoities. And it is this common
  substratum, this empty form, which is the basic intuition of
  mathematics.

