But do we really want to assume that arrows in [the[in the small-enough categories] are always pure sets? Isn't category theory supposed to be a story about how different bits of the mathematical universe hang together which doesn't presuppose some over-arching, all-in, set-theoretic reductionism, and so in particular doesn't presuppose from day one that all morphisms are pure sets??
Now, the foundational sections you often meet early in category theory oftenbooks worry away about questions of size (sets vs classes etc.). But the present worry is orthogonal to all that, and is in a way more basic. If we want to make no assumption that the denizens of different bits of the mathematical universe are all cut from the same cloth, we won't want to slip into assuming that sets of these denizens are all pure sets. So in particular, do we really want to assume that a collection of arrows (hom-set) must live in $\mathbf{Set}$ -- where that's the category mentioned back almost on p.1 of the book -- (as opposed, perhaps, to being fully faithfully mappable into that world?