I didn't have time to write up a proper answer on initially seeing your question and much of what I have to say has been said in the comments and Tim's answer, but I'll still offer some specifics on things mentioned in the comments.

The definition of a set is, as Tim Campion pointed out, the axioms of whatever set theory you're working in.  They do not so much define a set as describe a primitive notion that behaves like what we think sets should behave like, with different axiomatizations giving rise to differing notions of set.

These differing axiomatizations do have an impact on category theory because they change the behavior of ${\bf Set}$, and consequently the behavior of presheaf categories, and they also impact our ability to manipulate large categories. As mentioned by Ingo Blechschmidt in the comments, Mike Shulman has an excellent [paper](https://arxiv.org/abs/0810.1279) surveying some of these consequences. I will summarize some of them here, but I highly recommend you check out his paper.

A striking [result](https://doi.org/10.2307/2275291) referenced in the Shulman paper is due to Colin McLarty, establishing that the NF axiomatization of what a set is yields a ${\bf Set}$ that isn't Cartesian closed.

In ZFC we really only run into issues if we want to manipulate large categories as a whole, for example ${\bf Set}$ or ${\bf Group}$, which are not actually objects in ZFC since they're proper classes. We can get around this with shenanigans about formulas the metalanguage, but anyone looking for an integrated and 'natural' treatment of large categories on level footing with small ones will be disappointed in this setting.

[NBG](https://ncatlab.org/nlab/show/NBG) is a conservative extension of ZFC (meaning it doesn't prove anything about sets that ZFC can't) which does allow proper classes to be real objects in the theory, but we still run into some discomfort when dealing with large categories. NBG manages to be conservative over ZFC by restricting it's [comprehension axiom](https://ncatlab.org/nlab/show/axiom+of+separation) to only apply to sets, not proper classes -- in practical terms, as Mike points out in his paper linked above, this means (for example) that we can't prove by induction that a large category $\mathcal{C}$ has an $n$-fold Cartesian product $\mathcal{C}^n$. We can get around this by constructing it directly as the category of functions from $n$ into $\mathcal{C}$, but the unavailability of canonical proof methods like induction is troubling.

[MK](https://en.wikipedia.org/wiki/Morse–Kelley_set_theory) is a non-conservative extension of ZFC, essentially NBG but with full class comprehension allowed so we have access to all of the standard proof tools for large categories. This new theory can prove things ZFC can't, like the consistency of ZFC, and is thusly strictly stronger in a meaningful sense. MK also has its own serious issues when working with large categories -- we can't define the category of functors between two large categories, and this applies to NBG as well.

Using full MK further suggests that we try to look at the category of classes, since they're *really* the category of collections we want to work with right? And bam, once again we're back to a situation where we have to play games in the metalanguage, or conservatively extend/step up the consistency strength of our theory. This leads mathematicians to situations like [Grothendieck universes](https://ncatlab.org/nlab/show/Grothendieck+universe), where it's always possible to step up to the next universe if we need to talk about 'all the somethings' in the current universe. This is equivalent to working in ZFC plus an axiom asserting the existence of an [inaccessible cardinal](https://ncatlab.org/nlab/show/inaccessible+cardinal).

All the extra baggage of universes or inaccessibles is still somewhat of a sledgehammer for the problem at hand, though; all we want is for large categories to 'be like small categories' in enough ways that we can carry out all the constructions we care about with large categories, but inaccessibles or Grothendieck universes also have a plethora of other consequences (like the need to juggle universes<sup>1</sup>). A solution to these problems comes in the form of [reflection principles](https://ncatlab.org/nlab/show/reflection+principle), which are essentially axioms asserting that proper classes look enough like sets that we don't have to soil ourselves when they appear, but don't endow them with enough independence to give rise to a whole hierarchy of universes we need to ask questions about. All of this is discussed at length in the Shulman paper referenced above, with additional references therein.

First paper: Shulman, Mike. Set theory for category theory. [arXiv:0810.1279v2](https://arxiv.org/abs/0810.1279) [math.CT]

Second paper: McLarty, Colin. Failure of Cartesian Closedness in NF. J. Symbolic Logic 57 (1992), no. 2, 555--556. https://doi.org/10.2307/2275291

---

<sup>1</sup>As Tim points out in the comments, how many universes we have to juggle when taking this route is up to us. Skilled jugglers may use an infinite number, while those new to the approach may use only two.