I don't know any published work by Grothendieck specifically on modular forms.

It seems however that Grothendieck has spent some time thinking about moduli spaces of elliptic curves, for example. He has written a very long manuscript "La longue marche à travers la théorie de Galois", which is about what is now called Teichmüller theory, if I'm not mistaken. At some point, he even seems to do some "explicit" computations... I don't know this text very well though, so I would appreciate the opinion of other people about it (note that the text isn't available anymore on the Grothendieck Circle website).

I would also say that there are at present many approaches of modular forms in the "Grothendieck style". Here I would mention the whole theory of automorphic forms, by Deligne, Langlands and many others, which is at the same time very abstract and very powerful.

I don't know any published work by Grothendieck specifically on modular forms.

It seems however that Grothendieck has spent some time thinking about moduli spaces of elliptic curves, for example. He has written a very long manuscript "La longue marche à travers la théorie de Galois", which is about what is now called Grothendieck-Teichmüller theory, if I'm not mistaken. At some point, he even seems to do some "explicit" computations... I don't know this text very well though, so I would appreciate the opinion of other people about it (note that the text isn't available anymore on the Grothendieck Circle website).

I would also say that there are at present many approaches of modular forms in the "Grothendieck style". Here I would mention the whole theory of automorphic forms, by Deligne, Langlands and many others, which is at the same time very abstract and very powerful.