Looking at the draft that was linked above, it's more clear what Kunen means. He is just saying that the informal "definition" of the natural numbers that you might think of in school is circular when examined closely. And it is, in the sense that you have to start with some undefined concept, be it "natural number", "finite set", "proof", etc., to capture finiteness.
However, Kunen does not dwell on that sort of philosohical point. He is simply saying that there is a formal and non-circular definition of ω in set theory, as the smallest infinite ordinal. This does give a rigorous definition, but it doesn't ensure that "finite" in an aribitrary model corresponds to our actual notion of finite. That is something that cannot be ensured in first-order logic.