MathOverflow will be down for maintenance for approximately 3 hours, starting Monday evening (06/24/2013) at approximately 9:00 PM Eastern time (UTC-4).

3 mention possibility 2.3; added 37 characters in body

Off-hand, I can think of at least two reasons why a collection whose elements are all sets might fail to be a set:

1. The collection is too big (which you've mentioned)

2. The collection is too complicated. This is the case, for example, withgeneric :

1. Generic filters -- for $M$ a model of ZFC, an $M$-generic filter is not an $M$-set even though every element of the filter is an $M$-set and the filter itself has the same cardinality as an $M$-set (and is therefore not "too big").It is also the case when the set codes

2. Sets which code information that the set theory "ought not know" -- for example, zero sharp is a subset of the constructible universe, yet not a set of $L$ (nor adjoinable to $L$ by forcing) because it contains information $L$ cannot know.

3. Sets introduced by compactness or ultraproduct arguments. For example, an ultrapower of a model of ZFC taken with a nonprincipal ultrafilter will satisfy the axiom of regularity, yet will contain ill-founded sets. This is possible because the order type of any infinite descending $\in$-chain is not a set (a phenomenon which was first explained to me here).

Aside from that, there is also the case you mention of "obviously" ill-founded sets (that is, ill-founded sets whose $\epsilon$-chain is also a set), which are simply excluded by fiat.

2 added 48 characters in body; deleted 48 characters in body
1

Off-hand, I can think of at least two reasons why a collection whose elements are all sets might fail to be a set:

1. The collection is too big (which you've mentioned)

2. The collection is too complicated. This is the case, for example, with generic filters -- for $M$ a model of ZFC, an $M$-generic filter is not an $M$-set even though every element of the filter is an $M$-set and the filter itself has the same cardinality as an $M$-set (and is therefore not "too big"). It is also the case when the set codes information that the set theory "ought not know" -- for example, zero sharp is a subset of the constructible universe, yet not a set of $L$ (nor adjoinable to $L$ by forcing) because it contains information $L$ cannot know.

Aside from that, there is also the case you mention of ill-founded sets, which are simply excluded by fiat.