What I'm leaving outside of this comment, since I've alrady ridiculed myself in front of people who actually work with toposestopoi and know this topic better than I do, is the following:
- mention accessible and locally presentable categories. Categories of sheaves are locally presentable elementary topoi. These are very nice categories that albeit being large can be described by a set (sometimes a group is infinite, but can be generated by a finite set; sometimes a category is a proper class, but there is a set of objects generating it).
- Work "on a relative base": sets are no more special than another topos, they're only easier to handle. Most of topos theory works the same way if you fix a "base" topos once and for all, say $\cal S$, and study the 2-category of "toposes"topoi over $\cal S$".
- Since a topos is a generalized space, it turns out that this procedure is no different from studying the category of space-maps over a prescribed base $X$ as opposed to the category of all maps with variable codomain. $Set$ being the terminal object of $\bf Topoi$, ${\bf Topoi}/Set$ is merely $\bf Topoi$.