In the opposite category of the category of sets, and of many algebraic categories, the only small objects are the empty set and the singleton. A conceptual reason for this is Freyd's (or Gabriel and Ulmer's?) theorem that it is impossible for a category and its opposite both to be locally presentable, unless they are both posets.
Indeed, if $A$ is a set with at least two elements, consider functions $f:\{0,1\}^\kappa\to A$ where $\kappa$ is some infinite cardinal. If $\lambda<\kappa$ then $\{0,1\}^\kappa$ may be viewed as a $\lambda$-cofiltered limit of all products of at most $\lambda$ of the copies of $\{0,1\}$. For $A$ to be $\lambda$-small in $\mathrm{Set}^{\mathrm{op}}$, we would have to be able to guarantee that $f$ depends on at most $\lambda$ coordinates in the domain.
Since the opposite of the category of sets is the category of complete atomic Boolean algebras (CABAs), we can also make this argument directly in there, where it amounts to the fact that there are elements in a coproduct of CABAs that do not come from any smaller sub-coproduct, since we can always take a join or a meet of elements from every term in the coproduct.