I am looking at the Yoneda lemma trying to see where the assumption of "locally small" really comes in. Obviously in order to define a functor to the category sets using $Hom$-spaces we need our $Hom$-spaces to be sets. However if we consider a enriched-category, enriched over some non-locally small monoidal category M, then for any element of the category, our $Hom$-sets give us a functor $Hom(A,-)$ to M.

In particlar, in the statement $$ Hom(Hom(-,A),F) \simeq F(A), $$ where $F$ is a set-valued functor, where does the assumption of "smallness" play a role.

In the answer to this question, it is stated that the category of sets can be replaced by any Grothendieck universe $U$. However, the definition of a Grothendieck universe assumes that $U$ is a set. Moreover, the enriched Yoneda lemma again assumes "smallness". So there must be some obvious reason people are careful this, a reason I unfortunately seem to be too obtuse to see. Could somebody enlighten me?