I take your question to be about what we might call the
structuralist perspective, the view that we specify
mathematical objects and structures by their defining
structural features, ignoring any internal or otherwise
irrelevant structure that an instantiation of the object
might exhibit. You perceive a tension between this view and
the pure theory of sets, in which every set carries its
hereditary $\in$-structure. You propose that the concept
of urelements---objects that are not sets but which can be
elements of sets---provide exactly what is needed to
implement the structuralist perspective, for because
urelements have no internal set-theoretic structure, there
would seem to be nothing to ignore. So the plan appears to
be for us to present the natural numbers as given
canonically by urelements and thereby hope to finesse any
need to engage the structuralist perspective directly.
But this strategy doesn't actually succeed, does it, since
someone might permute the urelements---swap two of them,
say---and thereby build a perfectly good copy of the
natural numbers, still made from urelements. If the
urelements were supposed to provide for you a canonical concept of
the natural numbers, then you would have a canonical number $5$, but which urelement will you say is the real number
$5$? Similarly, as you mention, we might swap the "dots" in your question.
So even when we build our structures from urelements, the structuralist issue still arises. But the point of having them, if I understand you correctly, was to avoid that issue.
Secondly, urelements are often described as distinct but
indistiguishable, each having all the same properties as
the others. But this is problematic, since an urelement $x$
is the only urelement that has the property of being $x$,
as well as the only element of $\{x\}$ and so on. Perhaps
that urelement is also my favorite urelement! Or perhaps it
was created first among all the urelements, whatever that
might mean, or perhaps it even does have a secret internal,
irrelevent mathematical but not set-theoretic structure
that is hidden from our knowledge and which remains
inaccessible to us. You might reply that all these are
features of urelements that you want to ignore---they are
irrelevant---but this would simply be admitting that you
haven't avoided the structuralist issue with urelements.
I take these issues to show that urelements don't actually
help us avoid the need to engage with the structuralist
perspective directly. We want to adopt the structuralist
view, and to specify our mathematical objects by their
defining structural features rather than by the essential
nature of their constituent objects.
The urelement concept arises naturally from two views in
naive set theory, first, the view that one must have some
objects before it is sensible to speak of sets of objects,
and second, the view that set theory is essentially a
supplemental theory, built on top of other mathematical
theories, providing assistance in theoretic argument. One
first has the natural numbers, for example, whatever they
are, and then one may consider sets of natural numbers and
sets of these sets and so on, and the same for real
numbers, and these sets assist with the original
mathematical analysis.
Set theorists quickly realized, however, that the
structuralist perspective allowed them to abandon any need
for the urelements---all the favorite mathematical
structures can be constructed out of pure sets. Set theory
proceeds in a pure, elegant development without urelements,
and set theorists adopt the structuralist perspective
wholesale. (What is a set, really? I don't care---but I
care about the structure of its $\in$-relations to the
other sets.) Even the urelements themselves can be
simulated by finding structural copies of them within the
pure set theory, just as we construct the integers and the
real field.
In this way, both of the naive views mentioned two
paragraphs back are overturned: the cumulative hierarchy of
sets arises from nothing, towering higher than we can
imagine, while providing the desired instances of all of
our favored mathematical structures. This is the sense in
which set theory unifies mathematics, by providing a common
forum in which we can view all other mathematical arguments
as taking place.
Lastly, let me mention that the idea of permuting
urelements gave rise to the earliest consistency proofs of
$\neg AC$. One begins with a model of ZFA, and then fixes a
group of permutations of the urelements, restricting to the
universe of sets that hereditarily respect that group
action. It can be arranged that the resulting symmetric
model satisfies $ZFA+\neg AC$, and so we arrive at models
without the axiom of choice. It was not known how to do
this in a pure set theory until Cohen introduced the
forcing technique. Nevertheless, the Jech-Sochor embedding
theorem shows that every initial segment of a permutation model of ZFA has a
copy as a permutation model of ZF, in the pure theory, in
which the iterated power set structure of the atoms is
respected up to that bound. This theorem therefore simultaneously redeems the early
approach to $\neg AC$ using urelements, while also showing
that the method was not necessary for that application.
Apologies for this long answer...
