In what sense is set theory a foundation for mathematics? To my mind (for what that is worth), there are at least three (somewhat) distinct senses in which set 'theory' (I put "theory" in scare quotes to indicate that the notion of being able to reduce the language of mathematics to the language of sets may not imply that there is a background theory of sets for mathematics) might be regarded as a 'foundation' for mathematics:
i) as a 'universal language' in which all of mathematics may be expressed (i.e. to which all of mathematics may be reduced), and, having that
ii) the class of all true sentences of that language (on the order of "true elementary number theory")
iii) a formal axiomatic theory such as ZFC, Kelly-Morse, etc.
By the incompleteness theorems, iii) seems not to be a viable candidate for such a foundation, and by Tarski's theorem regarding the definition of truth in formalized languages, ii) seems not to be a viable candidate, either. That seems to leave i) as the viable candidate, but on the other hand, that seems too little (after all, naive set theory, being inconsistent, derives all well-formed formulas (of its formal language) so the unrestricted Axiom of Comprehension may more properly be considered a formation rule, but then the naive set theory known as ideal set theory would properly consist of just one axiom, Extensionality, and that seems too little, too).
So, to ask in another, slightly different, way; in what sense should set theory be considered a foundation for mathematics?