Actually, modular curves for congruence groups have canonical models over number fields, not Q (there exist congruence subgroups other than $\Gamma _0(N)$!). They even have reasonably nice integral models. Moreover, the modular forms are sections of a sheaf defined on the canonical model, and the sheaf extends to the integral model. This has the following consequence: if a modular form has Fourier coefficients $a_n$ in a number field $K$ (so are sections the form is a section of the sheaf over $K$), then the $a_{n}$ have bounded denominators, i.e., lie in $d^{-1}\mathcal{O}_{K}$ for some $d$.
Modular curves for arithmetic groups also have models over number fields, but the last statement definitely fails for forms that don't come from congruence groups. So something goes wrong with the beautiful picture we have for congruence modular curves, but I've never understood exactly what. But However, this is another indication that the link to arithmetic is more tenuous in the noncongruence case, and helps explain why number theorists are mainly interested in the congruence case.
[This was written as a comment on Buzzard's answer, but the site wouldn't let me post it.it (too long).]
