Off-hand, I can think of at least two reasons why a collection whose elements are all sets might fail to be a set:
The collection is too big (which you've mentioned)
The collection is too complicated. This is the case, for example, with:
Generic filters -- for $M$ a model of ZFC, an $M$-generic filter is not an $M$-set even though every element of the filter is an $M$-set and the filter itself has the same cardinality as an $M$-set (and is therefore not "too big").
Sets which code information that the set theory "ought not know" -- for example, zero sharp is a subset of the constructible universe, yet not a set of $L$ (nor adjoinable to $L$ by forcing) because it contains information $L$ cannot know.
Sets introduced by compactness or ultraproduct arguments. For example, an ultrapower of a model of ZFC taken with a nonprincipal ultrafilter will satisfy the axiom of regularity, yet will contain ill-founded sets. This is possible because the order type of any infinite descending $\in$-chain is not a set (a phenomenon which was first explained to me here).
Aside from that, there is also the case you mention of "obviously" ill-founded sets (that is, ill-founded sets whose $\epsilon$-chain is also a set), which are simply excluded by fiat.
