I think Akhil may be right. I believe Grothendieck did say something along the lines of this quote, specifically in reference to Belyi's Theorem . My recollection is that Belyi proved this theorem without knowing that Grothendieck was interested in it, and in working out his theory of Dessin D'Enfants, Grothendieck found he needed this result, but couldn't prove it. He then discovered that Belyi had given a rather elementary proof (I'll hesitate to call it trivial myself, since I recall finding it pretty clever).
If anyone has a copy of Grothendieck's Esquisse D'un Programme, maybe the specific quote is in there? I don't seem to have an English copy on my laptop, and all of Grothendieck's writing has been removed from the Grothendieck Circle's webpage per Grothendieck's request. (Interestingly, Wikipedia says this request was made in a letter to Illusie in January 2010.) I don't immediately see such a quote in the French version.
Edit: Here is the English translation of a relevant passage from Esquisse d'un Programme due to Leila Schneps and Pierre Lochak, as it appears in London Math. Soc. Lecture Notes Series vol. 242 (pp. 254-255; around page 15 on Grothendieck's typewritten manuscript):
Every finite oriented map gives rise to a projective non-singular algebraic curve defined over $\overline{\mathbb{Q}}$, and one immediately asks the question: which are the algebraic curves over $\overline{\mathbb{Q}}$ obtained in this way -- do we obtain them all, who knows? In more erudite terms, could it be true that every projective non-singular algebraic curve defined over a number field occurs as a possible "modular curve" parametrising elliptic curves equipped with a suitable rigidification? Such a supposition seemed so crazy that I was almost embarrassed to submit it to the competent people in the domain. Deligne when I consulted him found it crazy indeed, but didn't have any counterexample up his sleeve. Less than a year later, at the International Congress in Helsinki, the Soviet mathematician Bielyi announced exactly that result, with a proof of disconcerting simplicity which fit into two little pages of a letter of Deligne -- never, without a doubt, was such a deep and disconcerting result proved in so few lines!
In the form in which Bielyi states it, his result essentially says that every algebraic curve defined over a number field can be obtained as a covering of the projective line ramified over the points $0, 1$ and $\infty$. This result seems to have remained more or less unobserved. Yet it appears to me to have considerable importance. To me, its essential message is that there is a profound identity between the combinatorics of finite maps on the one hand, and the geometry of algebraic curves defined over number fields on the other. This deep result, together with the algebraic-geometric interpretation of maps, opens the door onto a new, unexplored world -- within reach of all, who pass by without seeing it.