For a general category the subobjects do indeed not have to form a set.
In the context of MacLane/Moerdijk you only look at toposes and there one has a natural isomorphism $Sub_{\mathbf{C}}(D) \cong Hom_{\mathbf{C}}(D,\Omega)$, where $\Omega$ is the subobject classifier.
So it follows from the axioms of a topos, (edit, thanks Mike:) if it is locally small, that $Sub_{\mathbf{C}}(D)$ is a set. When you prove that the basic examples, sheaves, finite sets, products of those, etc. are toposes you exhibit an object $\Omega$ and establish the above bijection. Before that point the left hand side could a priori be a proper class but the right hand side is a set, since you know that your category is locally small, and your bijection then shows that subobjects form a set.
Knowing this you can conclude that objects in full subcategories (edit, thanks again:) whose embedding preserves monos (e.g. if they are reflective) of toposes also have a set of subobjects, e.g. in all locally presentable categories...