In the opposite category of the category of sets, and of many algebraic categories, the only small objects are the empty set and the singleton. A conceptual reason for this is Freyd's (or Gabriel and Ulmer's?) theorem that it is impossible for a category and its opposite both to be locally presentable, unless they are both posets. 

Indeed, if $A$ is a set with at least two elements, consider functions $f:\{0,1\}^\kappa\to A$ where $\kappa$ is some infinite cardinal. If $\lambda<\kappa$ then $\{0,1\}^\kappa$ may be viewed as a $\lambda$-cofiltered limit of all products of at most $\lambda$ of the copies of $\{0,1\}$. For $A$ to be $\lambda$-small in $\mathrm{Set}^{\mathrm{op}}$, we would have to be able to guarantee that $f$ depends on at most $\lambda$ coordinates in the domain.