If i may add (although this post is kind of old)
This circularity "problem" (if one wishes to see it as such), appears in (what is refered as) classical mathematics.
Now one should bear in mind that even these classical mathematics (continuing along the lines of aristotle, euclid, archimedes, leibniz, cantor, hilbert, russel, goedel, etc..), have been cutoff and formalised (or sterilised if you like) to a greater extened that originaly meant.
In any case this is not the main argument.
But i would like to draw attention to the intuitionistic flavor of mathematics (and especially of the LEJ Brouwer path) (see for example LEJ Brouwer, Cambridge Lectures on Intuitionism, most of first lecture plus the appendix on marxists.org).
There Brouwer, aware of the problem, explicitly takes on the issue of mathematics over language or syntax.
Excerpt:
FIRST ACT OF INTUITIONISM
Completely separating mathematics from mathematical language and hence
from the phenomena of language described by theoretical logic,
recognising that intuitionistic mathematics is an essentially
languageless activity of the mind having its origin in the perception
of a move of time. This perception of a move of time may be described
as the falling apart of a life moment into two distinct things, one of
which gives way to the other, but is retained by memory. If the twoity
thus born is divested of all quality, it passes into the empty form of
the common substratum of all twoities. And it is this common
substratum, this empty form, which is the basic intuition of
mathematics.