Off-hand, I can think of at least two reasons why a collection whose elements are all sets might fail to be a set:

1. The collection is too big (which you've mentioned)

2. The collection is too complicated.  This is the case, for example, with:

  1. [Generic filters][1] -- for $M$ a model of ZFC, an $M$-generic filter is not an $M$-set even though every element of the filter is an $M$-set and the filter itself has the same cardinality as an $M$-set (and is therefore not "too big").

  2. Sets which code information that the set theory "ought not know" -- for example, [zero sharp][2] is a subset of the constructible universe, yet not a set of $L$ (nor adjoinable to $L$ by forcing) because it contains information $L$ cannot know.

  3. Sets introduced by compactness or ultraproduct arguments.  For example, an ultrapower of a model of ZFC taken with a nonprincipal ultrafilter will satisfy the axiom of regularity, yet will contain ill-founded sets.  This is possible because the order type of any infinite descending $\in$-chain is not a set (a phenomenon which was first explained to me [here][3]).

Aside from that, there is also the case you mention of "obviously" ill-founded sets (that is, ill-founded sets whose $\epsilon$-chain is also a set), which are simply excluded by fiat.


  [1]: http://en.wikipedia.org/wiki/Generic_filter
  [2]: http://en.wikipedia.org/wiki/Zero_sharp
  [3]: https://mathoverflow.net/questions/14622/how-can-an-ultrapower-of-a-model-of-zfc-be-ill-founded-yet-still-satisfy-zfc