I understand the question to be about how to represent syntax internally in a theory. The traditional approach in logic is to use natural numbers and Gödel encodings.
Computer science, and in particular programming language theory, have a very rich theory of syntax. The abstract view of syntactic expressions is that they are finite trees, and that is how syntax is actually represented internally in compilers and other programs that process syntactic expressions. Of course, trees can easily be dealt with in a set theory, so you could use them to present syntax.
(A word of warning: if you speak to a computer scientit and state that syntax is about sequences of symbols they'll think you ignorant, or that you teleported from 1950's. Concrete syntax that humans use is indeed made of sequences of symbols, but these are viewed as convenience for humans, and are prompty parsed into abstract syntax trees. Nobody ever writes code that processes syntax by working directly with sequences of symbols.)
Category theorists have something to say about syntax as well. Joyal's arithmetic universes were designed to give an abstract account of Gödel's incompleteness proofs. Joyal never managed to publish this work, but Emilia Maietti kindly wrote it up in Joyal's arithmetic universes via type theory.
This is not the end of the story however. (Some) computer scientists are obsessed with finding the best way to deal with bound variables and binding operators (such as $\forall$, $\exists$, $\int$, etc.). There are abstract mathematical accounts of syntax with binding, for example higher-order abstract syntax. Perhaps the most interesting to this audience is nominal syntax which uses permutation models of ZF. Murdoch Gabbay's publications page has a wealth or resources, perhaps one can start with Foundations of nominal techniques: logic and semantics of variables in abstract syntax (published as https://doi.org/10.2178/bsl/1305810911 in the Bulletin of Symbolic Logic).