Off-hand, I can think of at least two reasons why a collection whose elements are all sets might fail to be a set:
The collection is too big (which you've mentioned)
The collection is too complicated. This is the case, for example, with generic filters -- for $M$ a model of ZFC, an $M$-generic filter is not an $M$-set even though every element of the filter is an $M$-set and the filter itself has the same cardinality as an $M$-set (and is therefore not "too big"). It is also the case when the set codes information that the set theory "ought not know" -- for example, zero sharp is a subset of the constructible universe, yet not a set of $L$ (nor adjoinable to $L$ by forcing) because it contains information $L$ cannot know.
Aside from that, there is also the case you mention of ill-founded sets, which are simply excluded by fiat.