Personally I found the language of sets and classes confusing, just as you describe. I've never been sure precisely what operations on classes are allowed. For instance some textbooks mention the category of all functors from Set to Set as an example of a category which isn't locally small; but it seems to me that it's not a category at all, because its collection of objects is too large to even be a class.
I'm much happier with the formalism of Grothendieck universes and the universe axiom: Every set is contained in some Grothendieck universe. Typically we choose a universe U and agree that Set denotes the category of elements of U (or sets whose cardinality is an element of U). Then there's no problem in forming the functor category [Set, Set], and using ordinary set theory we can see that its Hom sets are indeed too big to be elements of U.
For related discussion see my question here.
