What are the conjugacy classes of the category of ($\kappa$-small) sets? $\newcommand{\unsim}{{\sim}}$The set of conjugacy classes of a group $G$ is the quotient of $G$ by the equivalence relation $\sim_1$ obtained by declaring $a\sim_1b$ if there exists some $g\in G$ such that $b=g^{-1}ag$. This agrees also with the equivalence relation

*

*$\unsim_2$ given by declaring $a\sim_2b$ if there exists some $g\in G$ such that $ga=bg$;

*$\unsim_3$ generated by $ab\sim_3 ba$.

Passing from groups to monoids, each of the above relations continue to make sense with a feel modifications; given a monoid $M$, we define $\unsim_1$, $\unsim_2$, and $\unsim_3$ to be the equivalence relation generated by (taking the symmetric and transitive closures of) the relations

*

*$\unsim'_1$ declaring $a\sim'_1 b$ if there exists some invertible $g\in M$ such that $a=g^{-1}bg$;

*$\unsim'_2$ declaring $a\sim'_2 b$ if there exists some $m\in M$ such that $ma=bm$;

*$\unsim'_3$ declaring $ab\sim'_3 ba$.

Each of the three relations above leads to distinct notions of conjugacy classes for monoids. They also admit the following category-theoretic descriptions:
\begin{align*}
M/\unsim_1 &\cong \mathsf{Fun}(\mathrm{B}\mathbb{N},\mathrm{B}M)/\{\text{isos}\},\\
M/\unsim_2 &\cong \pi_0(\mathsf{Fun}(\mathrm{B}\mathbb{N},\mathrm{B}M)),\\
M/\unsim_3 &\cong \int^{A\in\mathrm{B}M}\mathrm{Hom}_{\mathrm{B}M}(A,A).
\end{align*}
(The end $\int_{A\in\mathrm{B}M}\mathrm{Hom}_{\mathrm{B}M}(A,A)$ is also a familiar notion: it is the centre of $M$.)
More generally, we may replace $\mathrm{B}A$ with an arbitrary category $\mathcal{C}$, leading to three sensible notions of "conjugacy classes of categories". Similarly, we also have notions of

*

*"categories of conjugacy classes of monoidal categories$^\dagger$ and $2$-categories";

*"$\infty$-groupoids of conjugacy classes of $\infty$-categories";

*"$\infty$-categories of conjugacy classes of monoidal $\infty$-categories";

*and so on.

$^\dagger$For instance, given a monoidal category $\mathcal{C}$, we may define its category of "$\sim_3$-conjugacy" classes by first delooping it into a bicategory $\mathrm{B}\mathcal{C}$ and then taking the pseudo-bicoend of $\mathsf{Hom}_{\mathrm{B}\mathcal{C}}(-,-)$. (Again, the pseudo-biend is also an interesting object: it is the Drinfeld centre of $\mathcal{C}$.)

Main Question. Let $\kappa$ be a cardinal. Is there a nice(-ish) description of the set
$$\mathrm{Cl}(\mathsf{Sets}_{\leq\kappa})\cong\int^{X\in\mathsf{Sets}_{\leq\kappa}}\mathrm{Hom}_{\mathsf{Sets}_{\leq\kappa}}(X,X)$$
of ($\unsim_3$-)conjugacy classes of the category $\mathsf{Sets}_{\leq\kappa}$ of sets of cardinality $\leq\kappa$? In particular, what are the answers for the cases $\kappa=\aleph_0$ and $\kappa=2^{\aleph_0}$?
Also, what about the category
$\int^{\mathcal{C}\in\mathsf{Cats}}_{\mathsf{ps}}\mathsf{Fun}(\mathcal{C},\mathcal{C})$
of conjugacy classes of ("appropriately small"; e.g. finitely generated) categories, or the $\infty$-groupoid
$\int^{X\in\mathcal{S}}\mathrm{Map}(X,X)$ of conjugacy classes of ("appropriately small"; e.g. "$\pi$-finite") $\infty$-groupoids?
 A: So for endonorphisms up to isomorphisms, you're asking for a description of endomorphisms of sets. It sort of depends what kind of description you're looking for, but you could imagine something like "a decomposition $X = Y \sqcup Z$ and a surjection $X\to Y$". I'm not sure there are much simpler "classifications", because you need to classify surjections in a sense...
2 will be essentially the now deleted answer of Alexander, namely you idenfity all $g,f$ such that for some $u, u\circ f = g\circ u$. But any $f$ can be related that way to the unique endomorphism of the point, and so you only get a point in your $\pi_0$, which is then just a singleton.
For 3, the coend computation is very interesting - it was really fun to work this out !
First, note that we must to some extent restrict the co-end to make sure it exists. And indeed, Tom Goodwillie made the following observation in the comments : if $f,g$ are any two functions such that both composites are defined, then $f\circ g$ and $g\circ f$ have the same number of fixed points, so that the map from the coend to the class of ordinals defined by "number of fixed points" is well-defined, and surjective : if you want your coend to be a set, you'll need to restrict it somehow.
So say we restrict to finite sets for instance - this seems the more natural candidate ("compact objects"), but the amswer is also simpler - I'm not sure what it would be if you restricted to "countable sets" or something else. In particular the answer for $\infty$-groupoids will strongly depend on what kind of finiteness assumptions you impose. Let me not try to adress it here, I haven't thought about it for long enough.
For finite sets though, the main observation is the following : if $f$ is an arbitrary endomorphism, then you can write it as $i\circ p$, where $p$ is surjective and $i$ injective. Then it gets identified with $p\circ i$ in the co-end. Now if $f$ was not a bijection, then it was also not injective, and so $p\circ i$ is an endomorphism on a finite set of strictly lower cardinality.
In particular you can go down and down until you reach a bijection. It's easy to describe what this bijection actually is : it's the "eventual image" of $f$, namely you take the image of $f$, restrict $f$ to that, take the image of that and iterate until you reach a stable subset.
Then $f$ is a bijection thereon. Let's call this $im_\infty(f)$, and let us abuse notation by writing $f$ for the induced bijection. Now I want to say that you can't go further, the idea being that if you write a bijection as $g\circ f$, then $g$ is surjective and $f$ injective, so $f\circ g$ becomes this bijection after applying the previous $i\circ p \mapsto p\circ i$ transform - in particular you get no new identifications this way !
Then I claim that the map from the coend to $\pi_0((\mathrm{Fin}^\simeq)^{B\mathbb N})$ given by $f\mapsto (im_\infty(f), f)$ is a bijection.
1- it is well-defined. It's an easy induction to show that $f$ induces a map from $im_\infty(g\circ f)$ to $im_\infty(f\circ g)$, and conversely, and it's also not hard to show that it is in fact a bijection compatible with $f\circ g$ and $g\circ f$. In particular they define the same object in $\pi_0$ of the endomorphism groupoid of the groupoid of finite sets.
2- It is surjective. That much is clear from the fact that $im_\infty$ of a self-bijection of $X$ is $X$, with the same bijection.
3- it is injective. This is now also clear from the fact that I can always relate $(X,f)$ and $(im_\infty(f), f)$ by the above construction, so if $f$ and $g$ have the same $im_\infty$ (as sets with permutation) up to isomorphism, they can be related by such constructions too, and so must be equal in the coend.
In particular, the coend remembers the number of fixed points (i.e. the trace - this is reminiscent if Hochschild homology, which is the answer if you do this for projective modules over a ring for instance), but slightly more in fact, namely the whole cycle decomposition of the induced bijection on the eventual image.
Your other questions, about categories and $\infty$-groupoids sound interesting, but will most likely be complicated and/or depend on what restriction you want to impose on the index (for $\infty$-groupoids I can think of two reasonable ones : $\pi$-finite, or compact - for categories I'm not so sure)
