I didn't have time to write up a proper answer on initially seeing your question and much of what I have to say has been said in the comments and Tim's answer, but I'll still offer some specifics on things mentioned in the comments.
The definition of a set is, as Tim Campion pointed out, the axioms of whatever set theory you're working in. They do not so much define a set as describe a primitive notion that behaves like what we think sets should behave like, with different axiomatizations giving rise to differing notions of set.
These differing axiomatizations do have an impact on category theory because they change the behavior of ${\bf Set}$, and consequently the behavior of presheaf categories, and they also impact our ability to manipulate large categories. As mentioned by Ingo Blechschmidt in the comments, Mike Shulman has an excellent paper surveying some of these consequences. I will summarize some of them here, but I highly recommend you check out his paper.
A striking result referenced in the Shulman paper is due to by Colin McLarty, establishing that the NF axiomatization of what a set is yields a ${\bf Set}$ that isn't Cartesian closed.
In ZFC we really only run into issues if we want to manipulate large categories as a whole, for example ${\bf Set}$ or ${\bf Group}$, which are not actually objects in ZFC since they're proper classes. We can get around this with shenanigans about formulas the metalanguage, but anyone looking for an integrated and 'natural' treatment of large categories on level footing with small ones will be disappointed in this setting.
NBG is a conservative extension of ZFC (meaning it doesn't prove anything about sets that ZFC can't) which does allow proper classes to be real objects in the theory, but we still run into some discomfort when dealing with large categories. NBG manages to be conservative over ZFC by restricting it's comprehension axiom to only apply to sets, not proper classes -- in practical terms, as Mike points out in his paper linked above, this means (for example) that we can't prove by induction that a large category $\mathcal{C}$ has an $n$-fold Cartesian product $\mathcal{C}^n$. We can get around this by constructing it directly as the category of functions from $n$ into $\mathcal{C}$, but the unavailability of canonical proof methods like induction is troubling.
MK is a non-conservative extension of ZFC, essentially NBG but with full class comprehension allowed so we have access to all of the standard proof tools for large categories. This new theory can prove things ZFC can't, like the consistency of ZFC, and is thusly strictly stronger in a meaningful sense. MK also has its own serious issues when working with large categories -- we can't define the category of functors between two large categories, and this applies to NBG as well.
Using full MK further suggests that we try to look at the category of classes, since they're really the category of collections we want to work with right? And bam, once again we're back to a situation where we have to play games in the metalanguage, or conservatively extend/step up the consistency strength of our theory. This leads mathematicians to situations like Grothendieck universes, where it's always possible to step up to the next universe if we need to talk about 'all the somethings' in the current universe. This is equivalent to working in ZFC plus an axiom asserting the existence of an inaccessible cardinal.
First paper: Shulman, Mike. Set theory for category theory. arXiv:0810.1279v2 [math.CT]
Second paper: McLarty, Colin. Failure of Cartesian Closedness in NF. J. Symbolic Logic 57 (1992), no. 2, 555--556. https://projecteuclid.org/euclid.jsl/1183743976