Modular forms could be defined for arbitrary subgroups of the modular group, and I have read that this is done in some papers, but every definition of a modular form I have seen has been with respect to congruence subgroups.
4 Answers
At a certain level, it's mostly a matter of (i) terminology and (ii) reading the right books. Technically the word "modular" in modular forms refers to the "modular group $SL_2(\mathbb{Z})$".
In Miyake's book Modular Forms, he defines an automorphic form with respect to an arbitrary Fuchsian group $\Gamma$ (i.e., a discrete subgroup of $SL_2(\mathbb{R})$). Then he goes on to say (p. 114) that "Automorphic functions and forms for modular groups are called modular functions and modular forms respectively." Despite the title, plenty of the book deals with the general case, or with the special case of Fuchsian groups associated to quaternion algebras, which do not yield modular forms according to his definition.
In Shimura's book Introduction to the Arithmetic Theory of Automorphic Functions he defines (pp. 2829) automorphic functions and forms with respect to an arbitrary Fuchsian group of the first kind (i.e., finite hyperbolic covolume). The phrase "modular forms" is sometimes used in his book, but doesn't appear to get a formal definition.
These are, to my mind, the two most standard and authoritative references on "modular forms", and they both entertain the concept of a modular form with respect to a rather general Fuchsian group, whatever they want to call it.
On the other hand, there are reasons for restricting to Fuchsian groups which are arithmetic (which is a technical term here) and of congruence type. A theorem of Margulis shows that arithmeticity is equivalent to having a sufficiently rich theory of Hecke operators, which is highly important in numbertheoretic applications. Similarly, being arithmetic of congruence type puts you in the realm of Shimura varieties, and gives you a rich theory of models of the Riemann surfaces defined over various abelian number fields. On the other hand, it is a famous consequence of Belyi's theorem that every algebraic curve over $\mathbb{Q}$ can be uniformized by a finiteindex subgroup of $SL_2(\mathbb{Z})$ (generally of noncongruence type). So if one is interested in the "special" arithmetic properties of modular curves, it makes sense to restrict to congruence type.
Indeed, continuing work of John Voight and myself indicates that the congruence type condition is even more arithmetically significant than the arithmeticity [sic!]. We define congruence subgroups of nonarithmetic Fuchsian triangle groups and derive some of the arithmetic applications (using techniques from group theory and the arithmetic theory of branched coverings) that are parallel to those satisfied by the usual modular curves. See
http://alpha.math.uga.edu/~pete/triangle091309.pdf
Note that this work is not yet finished, to my consternation. (Mea culpa. Mea culpa.)
The main point is that the basic definitions work fine but the link with arithmetic is much more "vague". Look at early papers of Tony Scholl. There are Galois representations attached to certain noncongruence forms, but they will in general only be of a subgroup of the absolute group of Q and they're typically not 2dimensional. So in summary, you can study them, sure, as Scholl and others did, but there are fewer applications.
Here's an outline of what goes wrong. Modular curves defined by congruence subgroups have natural models over Q. A theorem of Eichler and Shimura says that the Tate module of the Jacobian of such a curve breaks up, as a Galois module, into "2dimensional bits". In the noncongruence case the curves aren't defined over Q and the galois representation on the Tate module doesn't break up into 2d pieces.
Actually, modular curves for congruence groups have canonical models over number fields, not Q (there exist congruence subgroups other than $\Gamma _0(N)$!). They even have reasonably nice integral models. Moreover, the modular forms are sections of a sheaf defined on the canonical model, and the sheaf extends to the integral model. This has the following consequence: if a modular form has Fourier coefficients $a_n$ in a number field $K$ (so the form is a section of the sheaf over $K$), then the $a_{n}$ have bounded denominators, i.e., lie in $d^{1}\mathcal{O}_{K}$ for some $d$.
Modular curves for arithmetic groups also have models over number fields, but the last statement definitely fails for forms that don't come from congruence groups. So something goes wrong with the beautiful picture we have for congruence modular curves, but I've never understood exactly what. However, this is another indication that the link to arithmetic is more tenuous in the noncongruence case, and helps explain why number theorists are mainly interested in the congruence case.
[This was written as a comment on Buzzard's answer, but the site wouldn't let me post it (too long).]

1$\begingroup$ Even away from Gamma_0(N) you can get models over Q; you just need to be more devious. For Gamma_1(N) the trick is to use the compact open of the adelic points defined by (* *;0 1) mod N rather than (1 *;0 1) (reflex field still Q and model is geom connected), and for Gamma_N (full level N) you can use Isom(Z/N x mu_N,E[N]) rather than Isom(Z/N x Z/N,E[N]) (although this probably isn't canonical in the sense of Delignenot sure). I don't know if there is definitely a congruence subgroup of SL_2(Z) for which the corresponding modular curve really has no model over Q. $\endgroup$ Apr 17, 2010 at 21:11

1$\begingroup$ Certainly not canonical. For example, the curve for Gamma_N classifies elliptic curves with a level N structure. Such objects have a discrete invariant, namely, an Nth root of 1 given by applying the Weil pairing to the basis. To get a connected family, you need to specify the Nth root, so this only makes sense over a field where you have one. Of course, for nonconnected curves, everything is defined over Q. $\endgroup$– JS MilneApr 17, 2010 at 21:55

$\begingroup$ @Kevin: yes, $\mathbb{Q}$ is a field of definition for every Riemann surface attached to a subgroup of $PSL_2(\mathbb{Z})$ of congruence type. $\endgroup$ Apr 17, 2010 at 23:49

1$\begingroup$ [I should perhaps say that it was of issues like this that I wrote "natural models over Q" rather than "canonical models" in my answer...]. @Milne: if one uses the full level N structure in the adelic definition then the variety won't be geometrically connected because the determinant map isn't surjective, and the canonical model is as you say. But there are other subgroups that one could use and I've never thought about them: for example can one use (* 0;0 1)? What does one get then? It's certainly geom connected and the intersection with PSL_2(Z) is just the classical congruence group. $\endgroup$ Apr 18, 2010 at 8:41

1$\begingroup$ @MarcPalm: Dear Marc, Your statement is not literally true as written. What is true is that there is a vector on which $\Gamma_0(p_v^N)$ acts through some nebentypus character (i.e. some character of the lower righthand entry, taken modulo $p_v^N$). In classical terms, this means that one can always work on $\Gamma_1(K)$ for $K$ large. This last statement is easy to see directly: the usual matrix $(0 \quad 1, N \quad 0)$ will conjugate $\Gamma(N)$ into $\Gamma_1(N^2)$. Regards, $\endgroup$– EmertonFeb 9, 2014 at 14:06
Another reason it is relatively uncommon to study automorphic cusp forms for noncongruence subgroups is that their existence is difficult if not impossible to establish, from the work of Phillips and Sarnak and others.
EDIT: In the above I was referring to functions on the upper half plane invariant under the group $\Gamma$ which are eigenfunctions of the hyperbolic Laplacian, i.e., Maass forms. I should have been more explicit! For the case of $\Gamma = SL_2(\mathbb{Z})$ their existence was only shown by Selberg using the trace formula. The trace formula holds for noncongruence subgroups too but in general the spectral side has both a discrete part (corresponding to cusp forms) and a continuous part (corresponding to Eisenstein series). For $\Gamma_0(N)$ one can show directly that the continuous part contributes a lower order of magnitude than the whole spectral side, and therefore deduce that Maass forms exist in abundance. For noncongruence subgroups this argument breaks down and the work referred to above provides evidence that the discrete spectrum is very small.

$\begingroup$ Take any noncongruence subgroup: take the corresponding quotient of the upper half planecompactify so you get a compact Riemann surface, probably of high genus; take a nonvanishing holomorphic differential, and voila! $\endgroup$ Apr 17, 2010 at 21:12

$\begingroup$ @Kevin: unless the genus of $\Gamma \backslash \mathcal{H}^*$ is $0$. Seriously, I've heard these claims about noncongruence subgroups before and always had a reaction like Kevin's. Moreover, it can't be that hard to generate cusp forms: if the number of cusps on the Riemann surface is N, then by pure linear algebra the codimension of the space of cusp forms in the space of weight k modular forms is at most N. So there have to be sufficiently many cusp forms. So what do people mean when they make these kind of claims? $\endgroup$ Apr 17, 2010 at 21:21

$\begingroup$ @Kevin: I see, you are claiming that any noncongruence subgroup of $\operatorname{PSL}_2(\mathbb{Z})$ has higher genus. Off the top of my head, I don't think that's true. Can you explain? $\endgroup$ Apr 17, 2010 at 21:23


$\begingroup$ @Matt: Ah, I see. Thanks very much for the explanation. $\endgroup$ Apr 17, 2010 at 23:21