When must it be  sets rather  than  proper classes, or vice-versa, outside of foundational mathematics? Every once  in a blue moon  it actually matters that some mathematical entity which might  a priori only be a class is in  fact  a set. For  clarification,  here are some examples of what I  do  not mean:  
A) Some colleagues of mine once  made the following disclaimer:  'The "set" of stable curves does not exist, but we leave this set theoretic difficulty to the reader.' These colleagues (names withheld to  protect the  innocent) are  of course  fully  aware of  the fact that strictly  speaking, the class of all  stable  curves, or topological  spaces, or  groups, or any the other usual suspects customarily formalized  in  terms of structured sets, cannot itself be  a set. While they recognize that  the  structure they study is transportable along arbitrary bijections between members of a proper class of equinumerous sets, they also recognize that in their setting this  same transportability could justify a  technically sufficient a  priori restriction to some fixed  but otherwise  arbitrary  underlying  set: that is, the relevant large category has a small skeletal  subcategory. (Exercise: precisely what makes this work  in the example given?) In such cases, the set versus class pecadillo is an essentially victimless  one, perhaps barring discussions  of the   admissibility of Choice, expecially  Global  Choice. 
B) There are various contexts in which seemingly unavoidable size issues are managed through the device of Grothendieck Universes. Such a move  beyond ZFC might  be regarded as cheating, sweeping the issue under the carpet for all the right reasons. Allegations of this nature regarding the use of derived functor cohomology in number theory, as in the proof  of Fermat's  Last Theorem, can  now be laid to rest, as Colin  McLarty has nicely  shown in "A finite order arithmetic foundation for cohomology" http://arxiv.org/abs/1102.1773.
C) Set  theory itself  is  replete with situations  where the set  versus class distinction is of paramount importance.  For just  one  example, my very  limited understanding  is  that forcing over  a  proper class of conditions is not  for the unwary. I'd be  interested to  hear some expert  elucidation of that, but my question  here is in  a  different spirit.   
With these nonexamples out of the way, I have a very short list of examples that do meet my criteria.  
1)  Freyd's theorem on the nonconcretizability of the homotopy category in "Homotopy  is  not concrete"  http://www.tac.mta.ca/tac/reprints/articles/6/tr6abs.html. By definition, a concretization of a  category is a faithful functor to the category of sets. The homotopy  category (of based  topological spaces) admits  no such functor.
The crux  of the  argument  is that  while  any object of a concretizable category has only a set's worth of generalized normal subobjects, there are objects  in the homotopy category - for example $S^2$ - which do not have this property (page  9). The original  closing remark (page 6)  mentions another nonconcretizability result, for the category of small categories and  natural equivalence classes of functors. A purist might try to disqualify the latter as too `metamathematical', but the homotopy example seems unassailable. 
2) A  category in  which all  (co)limits exist  is said to be (co)complete; a bicomplete category is one  which is both complete  and  cocomplete. Freyd's General Adjoint  Functor  Theorem gives necessary  and  sufficient conditions for the existence of adjoints  to  a  functor $\Phi:{\mathfrak A}\rightarrow{\mathfrak B}$ with $\mathfrak  A$ (co)complete. Let us say that  a  functor which  preserves all limits  is continuous, and that  one which preserves all colimits is cocontinuous.  A bicontinuous functor is one which is  both continuous and  cocontinuous.
Let us say that  $\Phi$ is  locally bounded  if for every $B\in  {\rm  Ob}\,{\mathfrak  B}$ there  exists a set $\Sigma$ such that for every $A\in{\rm Ob} \,{\mathfrak A}$ and  $b\in{\rm Hom}_{\mathfrak B}(B,\Phi A)$ there exist $\hat{A}\in{\rm Ob}\,{\mathfrak A}$ and $\hat{b}\in{\rm Hom}_{\mathfrak  B}(B,\Phi\hat{A})\cap\Sigma$ such that
$b=(\Phi \alpha)\hat{b}$ for some   $\alpha\in{\rm Hom}$ $_{\mathfrak A}$ $(\hat{A},A)$, and that $\Phi$  is  locally cobounded  if for every $B\in  {\rm  Ob}\,{\mathfrak  B}$ there  exists a set $\Sigma$ such that for every $A\in{\rm Ob}\,{\mathfrak A}$ and $b\in{\rm Hom}_{\mathfrak B}(\Phi A,B)$ there exist $\hat{A}\in{\rm Ob}\,{\mathfrak A}$ and $\hat{b}\in{\rm Hom}_{\mathfrak  B}(\Phi\hat{A},B)\cap\Sigma$ such that  $b=\hat{b}(\Phi \alpha)$ for some   $\alpha\in{\rm Hom}_{\mathfrak A}(A,\hat{A})$. In the literature these are known as the Solution Set Conditions.  
Theorem.
Let $\Phi:{\mathfrak   A}\rightarrow{\mathfrak B}$ be a  functor, where $\mathfrak  B$ is locally small.  
$\star$ If $\mathfrak A$ is complete then $ \Phi$ admits a left adjoint if and  only if  $\Phi$ is continuous and locally bounded.
$\star$ If $\mathfrak  A$ is cocomplete then $ \Phi$ admits a right adjoint if and  only if  $\Phi$ is cocontinuous and locally  cobounded.  
See pages  120-123 of MacLane's "Categories for the working mathematician".  
The local (co)boundedness condition has actual content. For  example:  
a) The  forgetful functor  ${\bf  CompleteBooleanAlgebra}\rightarrow{\bf Set}$ is continuous  but  admits  no left adjoint.  
b) Functors ${\bf Group}\rightarrow {\bf Set}$, continuous but  admitting no left  adjoint,  may be obtained as  follows: let $\Gamma_\alpha$   be a simple  group  of cardinality $\aleph_\alpha$ (e.g. the alternating group on a set of that cardinality,  or the projective special  linear group on a  2-dimensional vector space  over a field of that cardinality) and  take the   product (suitably construed), over  the  proper class  of all  ordinals, of the  functors  ${\rm  Hom}_{\bf Group}(\Gamma_\alpha,-)$.  
c) Freyd proposed another interesting example  (see page  -15 of the Foreword  to   "Abelian  categories" http://www.tac.mta.ca/tac/reprints/articles/3/tr3abs.html) of  a locally small bicomplete  category $\mathfrak S$ and a bicontinuous functor $\Phi:{\mathfrak S}\rightarrow {\bf Set}$ which  admits neither  adjoint: loosely speaking, the  category of  sets equipped with free  group  actions, and the evident underlying set functor. 
Does anyone know of any other examples, especially  fundamentally  different  examples?  
Finally, one could focus critical attention on the very question posed. To  what extent does the strength  and  flavor of the background  set theory  matter? Force  of habit and  comfort have me  implicitly working in some material set theory such as ZF, perhaps a bit more if I want to take  advantage of Choice, perhaps a bit less if I prefer to eschew Replacement. Indeed, I have actually checked that example b)  may be formulated in the absence of  Replacement: while the von  Neumann ordinals are no longer  available, the  same trick already used to give a kosher workaround to the illegitimate product  over all ordinals further shows that an appropriate system  of local ordinals suffices for the task. I am also quite interested in  hearing what proponents  of  structural  set theory have  to say.
 A: In my experience in analysis, basically the only place where it is actually important to distinguish sets from proper classes arises when one wishes to invoke Zorn's lemma to locate a maximal object in some non-empty partially ordered set $X$ in which all chains are bounded (e.g. to create a maximal proper subspace, a maximal filter, a maximally defined bounded linear functional, etc.).  Here it is crucial that $X$ is "small" enough to be an actual set (e.g. it is a collection of subsets of some space $V$ that is already known to be a set, or a collection of functions from $V$ to yet another set).  For instance, one cannot use Zorn's lemma to construct a maximal set in the class of all sets, or a maximal group in the class of all groups, or a maximal vector space in the class of all vector spaces, despite the fact that in each of these classes, any chain has an upper bound (the direct limit).  (Such maximal objects, if they existed, would soon lead to contradictions of the flavour of Russell's paradox or the Burali-Forti paradox; not coincidentally, one of the standard proofs of Zorn's lemma proceeds by contradiction, using the axiom of choice to embed all the ordinals into $X$, which can then be used to set up the Burali-Forti paradox.)
To put it another way: regardless of one's choice of foundations, it is clearly mathematically desirable to be able to easily locate maximal objects of various types; but it is obviously also desirable for the existence of such maximal objects to not lead (or mislead) one into paradoxes of Russell or Burali-Forti type.  ZFC, with Zorn's lemma on one hand and the set/class distinction on the other, manages to achieve both of these objectives simultaneously.  Presumably, many other choices of foundations (particularly those which are essentially equivalent to ZFC in a logical sense) can also achieve both objectives at once, but I usually don't see these points emphasised when such alternative foundations are presented in the literature.
A: Your question does not seemed aimed at set theorists, but let me
give a set theorist's answer.
I view the set/class distinction as analogous to and ultimately no
more problematic really than the other distinctions of size that
are commonly made in mathematics.
For example, we study the finite groups as a robust, coherent
collection, and we are untroubled by the fact that there are many than finitely many isomorphism types. We just don't find
it confusing that there are infinitely many finite groups. (For example, we 
don't expect to deduce by Zorn's lemma that there are maximal
finite groups.) Or we study the collection of countable graphs,
while realizing that there are uncountably many instances even on
the same set of vertices. More generally, we might look at
$\kappa$-dense topological spaces, or at all structures of a given
type of size less than a cardinal $\kappa$, or at spaces of a
given dimension or rank, and so on.
These distinctions of size are extremely common and part of the
way that we think mathematically; these distinctions are part of
the way that we carve up our mathematical universe at its joints.
Similarly, we may handle the set/class distinction, which is of
the same character, neither especially mysterious or problematic.
In each case, we have to pay attention to the details of the
mathematical constructions that we employ, in order that these
constructions not take us out of the class in focus.
As you say, set theory is replete with these considerations of
size and similar distinctions. The entire large cardinal hierarchy
is an investigation of different sizes of infinity. The
Grothendieck universe concept, arising at the entryway of that
hierarchy, is a such measure of size distinction, usually
considered a bit crude or clumsy by set theorists, but useful for
non-set-theorists because it is easy to understand. Meanwhile, set
theory is full of other subtler universe concepts: the levels of
the arithmetical and projective hierarchies provide "universes" of
complexity for countable objects; the various cut-off universes
$H_\kappa$, $L_\kappa$, $V_\kappa$ are often used as local
universe concepts; the proper-class sized inner models $L$,
$\text{HOD}$, $L(\mathbb{R})$, $L[0^\sharp]$ and so on provide
limitations of the background universe that is not just of "size",
but of set-theoretic complexity. In broad strokes, all these
limitations affect mathematical argument in a similar way, since
one must pay attention to which kinds of constructions might take
you beyond the limitation that has been set.
The set/class distinction is just one more such distinction. 
A: Terry Tao has already mentioned Zorn's Lemma in order to find maximal elements in small partial orders. More generally, colimits in categories usually only exist for small index categories. In fact, every category admitting colimits for all index categories is equivalent to a partial order.
Another typical example of this kind is the small object argument. It says that any set of morphisms in a category with certain conditions produces a functorial weak factorization system. The transfinite construction doesn't stop when we start with a class of morphisms.
Another example: A cocomplete symmetric monoidal category  is closed if and only if all functors $X \otimes -$ satisfy the solution condition. Todd Trimble has given an example where this fails. It is interesting that being closed is only a property of the data, but the property seems to depend on the size.
A: Proper classes come up when you exhaust the means of forming sets.   You need a set when you need to know the means of set theory have not been exhausted -- for example when you want to go on and form a colimit of the structures you have formed so far.   Exactly when the means are exhausted, depends on what means of forming sets you have.  
First take an example that exhausts second order arithmetic but does not exhaust Zermelo set theory (or simple type theory): the etale fundamental group of an arithmetic scheme.  There is no universal cover like the ones for topological spaces and this is not a logical or set theoretic problem but inherent in the situation.  (The scheme has etale covers of any finite degree, so a universal cover could have no finite degree.)  So Grothendieck and others formed the colimit of all symmetries of the (non-universal, actually existing) etale covers.    Second order arithmetic suffices to give the symmetry group of any one etale cover, but because we want the colimit of all these, we need an uncountable group.  Second order arithmetic will not produce that.   Third order will.
Grothendieck and Dieudonne often found they wanted colimits sort of like this, over all cases of some structure, but not just all that exist in second order arithmetic.  Naively put, they wanted all that exist in set theory.   Maybe all algebras over some ring, or all finitely generated algebras.  They knew there is a big difference between those examples, since there is not even a set of all algebras over a ring up to isomorphism (in any set theory they considered).  Choosing one countably infinite set of generators will give you a set of all finitely generated algebras over that ring up to isomorphism.  But in either case they did not want to bother with such details.  And they were all the more eager to avoid analogous details in more complex cases. 
If you really want to talk about all sets, or all natural weak equivalences of functors from Top to Top, or all generalized normal subobjects of $S^2$ in the homotopy category then you are exhausting the means of set theory (though the last two cases are less obvious than the first).  
Grothendieck and Dieudonne appreciated the point perfectly.   They knew workarounds to fit some of their larger constructions into ordinary set theory, and they were confident other workarounds could be found.  But they were not interested in that.  They saw that when they used all sets etc., it was not "all" in any metaphysical sense.  It was all those constructed by the ordinary means of set theory, so they posited one non-ordinary means of constructing sets: each set is contained in a universe.  At any point they work inside some universe, so what would be proper classes in ordinary set theoretic accounts are sets in the next larger universe.
