show/hide this revision's text 2 Added detail

Edit: I see I did not answer all your questions. I am not a logician, so take this with a grain of salt. But basically, in first order logic (in which ZFC is expressed) it is impossible to make circular definitions, and if you can't make one, you can't repair it. The circularity, as I see it, all exists on the meta-level, before you have even gotten around to formalizing the theory. So “breaking the circularity” must in essence happen in the transition between the informal and the formal.

Strictly speaking, first order theories don't even allow definitions at all! What you have to do is to notice that there is a complicated formula NN(x) that we interpret as “x is the set of natural numbers”, and a theorem ∃!x NN(x) in ZFC (where ∃! is short for “there exists a unique …”); then we create a new theory by adding the symbol ω and adding the axiom NN(ω). Now, any formula A(ω) in the new theory can be rewritten in the old theory as ∃x:NN(x)∧A(x), so nothing new has really happened, except for a great amount of simplification.
show/hide this revision's text 1

I don't have that book, but as far as I can understand, the “circularity” must mean this: in the phrase “iterating the successor operation a finite number of times”, we should mean a number of times corresponding to a natural number. But since the natural numbers are what we are defining, this is circular. So one has to define the natural numbers without reference to the concept of “finite”. Where the circularity is broken is if you rewrite your definition as follows:

  1. 0 is a natural number,
  2. the successor of any natural number is a natural number, and
  3. nothing is a natural number unless it must be, by 1 and 2.

(All this stated in more technical language, of course.)

There is no reference to the notion of “finite” here. Instead, number 3 above gives us, by definition, the principle of induction. For example, how do you show that some object X is not a natural number? Well, if for some property P, you can show P(0) and you can also show that ∀n: P(n)⇒P(n') where the prime denotes the successor function, but X fails property P, then you can know for sure that X is not a natural number.