To define a Grothendieck topology of a category, we usually require that the category is small.
Question 1: Why do we need to require the category to be small?
I thought that the problem was that we wanted a sieve to be a set. (In the category of manifolds, the maximal sieve of an arbitrary manifold $X$ is not a set.) But my thought must be wrong because of the following remark.
When we want to deal with the category of manifolds (schemes, or topological spaces), Metzler says that we can avoid this problem by choosing a subcategory which is small. For example, the category $\mathbf{M}$ of smooth manifolds which are second countable and Hausdorff.
Question 2: Why does this trick work?
Metzler explains the reason in Remark 13 in the linked paper, but I can not understand it (because I do not know what is the problem.)
Question 3: If we define a Grothendieck topology as an equivalence class of basis of Grothendieck topology, does this definition work?
In the definition of a basis of Grothendieck topology (See Def 5 in the paper by Metzler for the definition), we seem to deal with only sets. So I think that the definition is well-defined for any category.