The main point is that the basic definitions work fine but the link with arithmetic is much more "vague". Look at early papers of Tony Scholl. There are Galois representations attached to certain non-congruence forms, but they will in general only be of a subgroup of the absolute group of Q and they're typically not 2-dimensional. So in summary, you can study them, sure, as Scholl and others did, but there are fewer applications.
Here's an outline of what goes wrong. Modular curves defined by congruence subgroups have natural models over Q. A theorem of Eichler and Shimura says that the Tate module of the Jacobian of such a curve breaks up, as a Galois module, into "2-dimensional bits". In the non-congruence case the curves aren't defined over Q and the galois representation on the Tate module doesn't break up into 2-d pieces.
