For a general category the subobjects do indeed not have to form a set. 

In the context of MacLane/Moerdijk you only look at toposes and there one has a natural isomorphism $Sub_{\mathbf{C}}(D) \cong Hom_{\mathbf{C}}(D,\Omega)$, where $\Omega$ is the subobject classifier. 

So it follows from the axioms of a topos that $Sub_{\mathbf{C}}(D)$ is a set. When you prove that the basic examples, sheaves, finite sets, products of those, etc. are toposes you exhibit an object $\Omega$ and establish the above bijection. Before that point the left hand side could a priori be a proper class but the right hand side is a set, since you know that your category is locally small, and your bijection then shows that subobjects form a set.

Knowing this you can conclude that objects in full subcategories of toposes also have a set of subobjects, e.g. in all accessible categories...