My interpretation of Mazur's quote was in terms of the history of the discovery of modular forms. Of course trigonometric functions came first, and then a whole variety of other special functions in the eighteenth and nineteenth century, culminating in the study of elliptic functions. Since elliptic functions exhibit some kind of "universality" among special functions (most can be written in terms of evaluations of elliptic functions or limits thereof), when people first considered generalizations of their symmetries, it wasn't even clear if they would find anything.
This is also illustrated by the section "Poincaré on discovery and his work on automorphic functions" in the wiki article on automorphic forms:
One of Poincaré's first discoveries in mathematics, dating to the 1880s, was automorphic forms. He named them Fuchsian functions, after the mathematician Lazarus Fuchs, because Fuchs was known for being a good teacher and had researched on differential equations and the theory of functions. Poincaré actually developed the concept of these functions as part of his doctoral thesis. Under Poincaré's definition, an automorphic function is one which is analytic in its domain and is invariant under a discrete infinite group of linear fractional transformations. Automorphic functions then generalize both trigonometric and elliptic functions.
Poincaré explains how he discovered Fuchsian functions:
For fifteen days I strove to prove that there could not be any functions like those I have since called Fuchsian functions. I was then very ignorant; every day I seated myself at my work table, stayed an hour or two, tried a great number of combinations and reached no results. One evening, contrary to my custom, I drank black coffee and could not sleep. Ideas rose in crowds; I felt them collide until pairs interlocked, so to speak, making a stable combination. By the next morning I had established the existence of a class of Fuchsian functions, those which come from the hypergeometric series; I had only to write out the results, which took but a few hours.