Grothendieck was aware (at least) of modular forms and their relations to motives by J P Serre in a letter dated December 31, 1986.
Grothendieck (also) develop on "La longue Marche" a theory of "topos modulaire" (now completely available online!online!).