First, most mathematicians don't really care whether all sets are "pure" -- i.e., only contain sets as elements -- or not. The theoretical justification for this is that, assuming the Axiom of Choice, every set can be put in bijection with a pure set -- namely a von Neumann ordinal.
I would describe Bourbaki's approach as "structuralist", meaning that all structure is based on sets (I wouldn't take this as a philosophical position; it's the the most familiar and possibly the simplest way to set things up), but it is never fruitful to inquire as to what kind of objects the sets contain. I view this as perhaps the key point of "abstract" mathematics in the sense that the term has been used for past century or so. E.g. an abstract group is a set with a binary relation: part of what "abstract" means is that it won't help you to ask whether the elements of the group are numbers, or sets, or people, or what.
I say this without having ever read Bourbaki's volumes on Set Theory, and I claim that this somehow strengthens my position!
Namely, Bourbaki is relentlessly linear in its exposition, across thousands of pages: if you want to read about the completion of a local ring (in Commutative Algebra), you had better know about Cauchy filters on a uniform space (in General Topology). In places I feel that Bourbaki overemphasizes logical dependencies and therefore makes strange expository choices: e.g. they don't want to talk about metric spaces until they have "rigorously defined" the real numbers, and they don't want to do that until they have the theory of completion of a uniform space. This is unduly fastidious: certainly by 1900 people knew any number of ways to rigorously construct the real numbers that did not require 300 pages of preliminaries.
However, I have never in my reading of Bourbaki (I've flipped through about five of their books) been stymied by a reference back to some previous set-theoretic construction. I also learned only late in the day that the "structures" they speak of actually get a formal definition somewhere in the early volumes: again, I didn't know this because whatever "structure-preserving maps" they were talking about were always clear from the context.
Some have argued that Bourbaki's true inclinations were closer to a proto-categorical take on things. (One must remember that Bourbaki began in the 1930's, before category theory existed, and their treatment of mathematics is consciously "conservative": it's not their intention to introduce you to the latest fads.) In particular, apparently among the many unfinished books of Bourbaki lying on the shelf somewhere in Paris is one on Category Theory, written mostly by Grothendieck. The lack of explicit mention of the simplest categorical concepts is one of the things which makes their work look dated to modern eyes.