If $C$ is small, $C[W^{-1}]$ is again small, even without a calculus of fractions.  The problem in general is that when $C$ is a locally small large category, the zig-zag sequences can range over finite sequences of objects of $C$, which form a large set.

Essentially, the problem is that the hom-sets in the localization before taking quotients can, without additional assumptions, have the cardinality of the set of objects, and there may be sets of morphisms even after quotienting with cardinality equal to the cardinality of C.

This is a problem when C is large, since it means that the hom-sets can be large.