$\newcommand{\unsim}{{\sim}}$The set of conjugacy classes of a group $G$ is the quotient of $G$ by the equivalence relation $\sim_1$ obtained by declaring $a\sim b$ iff there exists $g\in G$ such that $b=g^{-1}ag$. This agrees also with the equivalence relations $\sim_2$ and $\sim_3$ on $G$ generated by [$ma\sim_2 bm$, $m\in G$] and $ab\sim_3 ba$.
These three equivalence relations make sense also for monoids, but now they are inequivalent notions: for a general monoid $M$, we can have $M/\unsim_1\ncong M/\unsim_2\ncong M/\unsim_3$. Each of these also admit nice descriptions in terms of category-theoretic notions: \begin{align*} M/\unsim_1 &\cong \mathsf{Fun}(\mathrm{B}\mathbb{N},\mathrm{B}M)/\{\text{isos}\},\\ M/\unsim_2 &\cong \pi_0(\mathsf{Fun}(\mathrm{B}\mathbb{N},\mathrm{B}M)),\\ M/\unsim_3 &\cong \int^{A\in\mathrm{B}M}\mathrm{Hom}_{\mathrm{B}M}(A,A). \end{align*} (The end $\int_{A\in\mathrm{B}M}\mathrm{Hom}_{\mathrm{B}M}(A,A)$ is also a familiar notion: it is the centre of $M$.)
Now, each of the above expressions makes perfect sense if instead of $\mathrm{B}A$ we started with a general category $\mathcal{C}$. This leads to three sensible notions of "conjugacy classes of a category", and there are also similar notions of "categories of conjugacy classes of monoidal categories$^\dagger$ and $2$-categories", "$\infty$-groupoids of conjugacy classes of $\infty$-categories", "$\infty$-categories of conjugacy classes of monoidal $\infty$-categories", and so on.
$^\dagger$E.g. given a monoidal category $\mathcal{C}$, we may define its category of "$\sim_3$-conjugacy" classes by first delooping it into a bicategory $\mathrm{B}\mathcal{C}$ and then taking the pseudo-bicoend of $\mathsf{Hom}_{\mathrm{B}\mathcal{C}}(-,-)$. (Again, the pseudo-biend is also an interesting object: it is the Drinfeld centre of $\mathcal{C}$.)
Is there a "nice(-ish)" description of the set $$\mathrm{Cl}(\mathsf{Sets})\cong\int^{X\in\mathsf{Sets}}\mathrm{Hom}_{\mathsf{Sets}}(X,X)$$ of ($\unsim_3$-)conjugacy classes of sets?
What about the category $\int^{\mathcal{C}\in\mathsf{Cats}}_{\mathsf{ps}}\mathsf{Fun}(\mathcal{C},\mathcal{C})$ of conjugacy classes of categories, or the $\infty$-groupoid $\int^{X\in\mathcal{S}}\mathrm{Map}(X,X)$ of conjugacy classes of $\infty$-groupoids?