By the modular group I mean either $SL(2,\mathbb{Z})$ or $PSL(2,\mathbb{Z})$.
Where can I find examples of these?
Another question: is there a good (ideally analytical, but possibly computer-aided) way to determine if a subgroup of the modular group generated by some given matrices has finite index, and possibly allowing to compute the index?
The answer to your Question 1. is that there are not rank 2 subgroups of $PSL_2(\mathbb{Z})$, but there are rank 3 subgroups. This follows from a result of Wohlfahrt. He shows in Theorem 5 that any non-congruence subgroup $\Gamma < PSL_2(\mathbb{Z})$ has index $\geq 7$. Misha points out in a comment to his answer that a finite coarea fuchsian group of rank $N$ has $Area(\mathbb{H}^2/\Gamma)\leq 2\pi(N-1)$. Therefore in the case $\Gamma$ is non-congruence, $Area(\mathbb{H}^2/\Gamma)=[PSL_2(\mathbb{Z}):\Gamma] Area(\mathbb{H}^2/PSL_2(\mathbb{Z})) \geq 7\pi/3$. Thus, $N=rank(\Gamma) \geq 3$. Misha's claim (in the non-uniform case) follows from an application of the free product decomposition of non-uniform Fuchsian group into cyclic groups, Grushko's theorem, and Gauss-Bonnet.
Wohlfahrt also gives an example of a non-congruence subgroup of index $7$ in $PSL_2(\mathbb{Z})$. On p. 531, he points out that there exists an index $7$ subgroup with two cusps. One determines that the group has genus $0$, and two cone points of orders $2$ and $3$ respectively. This group is therefore rank $3$, isomorphic to $\mathbb{Z}\ast \mathbb{Z}/2\ast\mathbb{Z}/3$. I didn't check, but I think this also lifts to a rank 3 subgroup of $SL_2(\mathbb{Z})$.
For your first question: "Most" 2-generated subgroups of the modular group will have infinite covolume, so they are not congruence subgroups. For instance, take the subgroup generated the upper and lower triangular matrices with off-diagonal entries equal to $2$.
For your second question: There is Jorgensen's algorithm for constructing fundamental domain for a subgroup of $PSL(2,\mathbb Z)$ with the given set of generators, see my answer in (un)decidability in matrix groups After you construct such fundamental polygon you can check if it has any edges contained in the circle at infinity of $\mathbb H^2$. This is equivalent to having infinite index. Analytically, finite index is equivalent to finiteness of covolume.
A. J. Scholl has written a series of articles on modular forms on noncongruence subgroups. In the article
On the Hecke algebra of a noncongruence subgroup, Bull. London Math. Soc. 29 (1997), no. 4, 395–399
he gives two examples of index 7 subgroups of $\mathrm{SL}_2(\mathbf{Z})$ generated by 3 elements :
$$\Gamma_{4,3} = \Bigl\langle \begin{pmatrix} 1 & 4 \\ 0 & 1 \end{pmatrix},\begin{pmatrix} 2 & 1 \\ 1 & 1 \end{pmatrix},\begin{pmatrix} 1 & -1 \\ 2 & -1 \end{pmatrix} \Bigr\rangle$$
$$\Gamma_{5,2} = \Bigl\langle \begin{pmatrix} 1 & 5 \\ 0 & 1 \end{pmatrix},\begin{pmatrix} 0 & -1 \\ 1 & 0 \end{pmatrix},\begin{pmatrix} 2 & 3 \\ 1 & 2 \end{pmatrix} \Bigr\rangle$$
The subscripts refer to the widths of the cusps (both groups have exactly two cusps).
I've compiled my serial comments on Misha's answer with this new answer, not containing any information unavailable in the other answers, but working out Wohlfart's example mentioned by Agol in more explicit detail. In particular there is more than one distinct conjugacy class (up to orientation reversal) of index 7 subgroups isomorphic to $\mathbb{Z} * \mathbb{Z}/3 * \mathbb{Z}/2$ and with two cusps; I have found three such. Extra information Wohlfart gives regarding amplitude of the cusps pins down which one it is.
The modular orbifold, of genus zero with one cusp, one $\mathbb{Z}/3$ cone point, and one $\mathbb{Z}/2$ cone point, deformation retracts to a graph of groups $G$ consisting of the unique geodesic connecting those two cone points. Covering spaces of the modular orbifold up to covering isomorphism (in the category of orbifolds), correspond to coverings of $G$ up to covering isomorphism (in the category of graphs of groups), the covering graph sitting as a deformation retraction of the covering orbifold. Each finite degree covering graph of $G$ is a finite, connected, bipartite graph of groups, where each vertex in one set is either of valence 2 labelled with the trivial group or of valence 1 labelled with $\mathbb{Z}/2$, and each vertex in the other set is either of valence 3 labelled with the trivial group or of valence 1 labelled with $\mathbb{Z}/3$; all edges are labelled with the trivial group. Its fundamental group is a free product of: $e_\infty$ copies of $\mathbb{Z}$, $e_3$ copies of $\mathbb{Z}/3$, and $e_2$ copies of $\mathbb{Z}/2$. The minimal number of generators equals $e_\infty + e_3 + e_2$, and $e_\infty = 2 \cdot $(genus)$ + $(number of cusps)$ - 1$. Also, Wolhfart gives a formula for index taken from a paper of Peterrson (which Agol alludes to in his answer), and which amounts to $$index = 6 \cdot e_\infty + 4 e_3 + 3 e_2 - 6 $$
Wolhfart's noncongruence example has index 7 with two cusps, one of amplitude 1 and one of amplitude 6. The amplitude is equal to half the number of edges of the edge path in the graph that is obtained from a circle around the cusp by projecting that circle to the graph under the deformation retraction. If one adds the restriction that this is the 3 generator example, one gets $e_\infty = e_2 = e_3 = 1$, and the corresponding hyperbolic orbifold is a twice punctured disc (an open annulus) with one $\mathbb{Z}/2$ cone point and one $\mathbb{Z}/3$ one point. If one requires the cusp at the origin to have amplitude 1 and the cusp at infinity to have amplitude 6, this is enough to pin down the graph in $\mathbb{C}-0$ uniquely (up to isotopy and orientation reversal).
First draw a circle around the origin, subdivided at two vertices into two edges. One of those vertices will have valence 2 in the graph, the other will have valence 3. Attach to the latter vertex an arc of two edges on the outside of the circle; the interior vertex of that arc will have valence 2, and its opposite vertex will have valence 3. Attach to that opposite vertex two arcs: one arc has one edge and opposite vertex of valence 1 labelled $\mathbb{Z}/2$; the other has two edges, interior vertex of valence 2, and opposite vertex of valence 2 labelled $\mathbb{Z}/3$.
$\DeclareMathOperator\PSL{PSL}\DeclareMathOperator\F{\mathbf{F}}$Let me provide a purely group-theoretic answer.
Let $H$ be a subgroup of finite index in $\mathrm{PSL}_2(\mathbf{Z})\simeq C_2\ast C_3$. Then, by Kurosh, $H$ is isomorphic to $C_2^{\ast e_2}\ast C_3^{\ast e_3}\ast \mathbf{Z}^{\ast e_0}$ for some integers $e_0,e_2,e_3\ge 0$ (clearly finite, and satisfying $e_2+e_3+e_0\ge 2$).
Since these are virtually free and since in a free group the index determines the rank, the triple $(e_2,e_3,e_0)$ determines the index $f(e_2,e_3,e_0)$ of $H$ in $\mathrm{PSL}_2(\mathbf{Z})$. Let me explicitly compute $f$.
For any groups $A,B$, we have $A\ast B\simeq A\ltimes B^{\ast A}$ (a kind of non-commutative wreath product). Hence, if $A'$ has index 2 in $A$, then $A\ast B$ has a subgroup of index 2 isomorphic to $A'\ltimes B^\ast {A'\sqcup A'}\simeq A'\ltimes (B\ast B)^{A'}\simeq A'\ast B\ast B$.
Given that $C_2^{e_2}$ has a subgroup of index 2 isomorphic to $\mathbf{Z}^{\ast e_2-1}$ for $e_2\ge 1$ (the kernel of the map onto $C_2$ that's identity on each factor), we deduce that $2f(e_2,e_3,e_0)=f(0,2e_3,2e_0+e_2-1)$ for $e_2\ge 1$.
Similarly, given that $C_3^{e_3}$, for $e_3\ge 1$ has a subgroup of index 3 isomorphic to $\mathbf{Z}^{\ast 2(e_3-1)}$, we deduce similarly that $3f(e_2,e_3,e_0)=f(3e_2,0,3e_0+2(e_3-1))$ whenever $e_3\ge 1$.
Hence $$f(e_2,e_3,e_0)=f(0,2e_3,2e_0+e_2-1)/2=f(0,0,6e_0+3(e_2-1)+2(2e_3-1))/6$$ $$=\frac16f(0,0,6e_0+3e_2+4e_3-5)$$ whenever $e_2,e_3\ge 1$; when one or both equals zero this is still valid, by direct verification and using that $f(0,0,1+k)=f(0,0,1+sk)/s$ by the values of ranks of subgroups of finite index in free groups.
Next, $\mathrm{PSL}_2(\mathbf{Z})$ has index 1 in itself, so $f(1,1,0)=1$. We just checked that $f(1,1,0)=f(0,0,2)/6$ by constructing a copy of index 6 of $F_2$. So $f(0,0,2)=6$, and, in turn, $f(0,0,n)=f(0,0,1+(n-1))=(n-1)f(0,0,2)=6(n-1)$. To conclude,
$$f(e_2,e_3,e_0)=\frac16f(0,0,6e_0+3e_2+4e_3-5)= 3e_2+4e_3+6(e_0-1).$$
When the generating rank is two, i.e., $e_0+e_2+e_3=2$ (excluding $e_2=2$ as $C_2^{\ast 2}$ can't occur), we get: $$f(1,1,0)=1,\;f(1,0,1)=3;\;f(0,1,1)=4;f(0,2,0)=2;f(0,0,2)=6.$$
Hence subgroups of finite index of generating rank 2 in $\mathrm{PSL}_2(\mathbf{Z})$ are precisely those subgroups of index $1,2,3,4,6$. In index 1,2 there is a single such subgroup.
So it remains to describe subgroups of index 1,2,3,4,6 in $\PSL_2(\mathbf{Z})$.
Index 1 ($\simeq C_2\ast C_3$): the whole group (= principal congruence modulo 1).
Index 2 ($\simeq C_3\ast C_3$): there is a single one. It is congruence modulo 2, since $\PSL_2(\F_2)$ is a dihedral group of order 6.
Index 3 ($\simeq C_2\ast\mathbf{Z}$ or $\simeq C_2^{\ast 3}$): up to conjugacy there are two such subgroups. One is normal, actually $\simeq C_2^{\ast 3}$: it is congruence modulo 3, since $\PSL_2(\F_3)\simeq\mathrm{Alt}_4$ has a normal subgroup of index 3. The other ones are non-normal and congruence modulo 2, since $\PSL_2(\F_2)$ has a non-normal subgroup of index 3. [One representative consists of those matrices that are $\pm I_2$ modulo 3. If I'm correct it has a single conjugacy class of elements of order 2, so this group is isomorphic to $C_2\ast\mathbf{Z}$.]
Index 4 ($\simeq C_3\ast\mathbf{Z}$ or $\simeq C_2^{\ast 2}\ast C_3$): up to conjugacy there are two such subgroups, if I'm correct. One is congruence modulo 3, since $\PSL_2(\F_3)\simeq\mathrm{Alt}_4$ has a subgroup of index 4 (of order 3). [This one has no element of order 2, hence is isomorphic to $C_3\ast\mathbf{Z}$.] One is congruence modulo $4$, since $\PSL_2(\mathbf{Z}/4\mathbf{Z})$, which has order 48, has a subgroup of index 4 (of order 12), if I checked correctly.
Index 6 ($\simeq\mathbf{Z}^{\ast 2}$ or $\simeq C_3^{\ast 3}$ or $\simeq C_2^{\ast 4}$) is the most interesting case. I'll develop this later as it requires a number of verifications.
This is an old question, but I wanted to add a sort of elementary answer, which shows in particular that any subgroup of $\operatorname{PSL}_2(\mathbb{Z})$ of index 7 isomorphic to $\mathbb{Z}*\mathbb{Z}/2\mathbb{Z}*\mathbb{Z}/3\mathbb{Z}$ cannot be a congruence subgroup. These subgroups were mentioned above, and are also fairly easy to construct by drawing a graph with two colors representing a covering space of $B\operatorname{PSL}_2(\mathbb{Z})$ (see here for a sketch of the method). We will use theorem 2 from the Wohlfahrt paper linked in Ian’s answer (which takes around one page to prove), together with the genus formula $g=1+\frac{d}{12}-\frac{\varepsilon_2}{4}-\frac{\varepsilon_3}{3}-\frac{\varepsilon_{\infty}}{2}$ where $\varepsilon_i$ is the number of elliptic points on $X(\Gamma)$ of order $i=2,3$, and $\varepsilon_{\infty}$ is the number of cusps.
Let $\Gamma$ be a subgroup of $\operatorname{PSL}_2(\mathbb{Z})$ of the proscribed form. Note that subgroups of $\operatorname{PSL}_2(\mathbb{Z})$ correspond to covering spaces of $B\operatorname{PSL}_2(\mathbb{Z})\simeq (B\mathbb{Z}/2\mathbb{Z})\vee (B\mathbb{Z}/3\mathbb{Z})$ (which is homotopy equivalent to $Y(1)$, at least as orbifolds, as in Lee’s answer). You can picture these covering spaces as here.
The covering space corresponding to $\Gamma$ is homotopy equivalent to $Y(\Gamma)$ (as orbifolds), and by looking at the fibers over $B\mathbb{Z}/2\mathbb{Z}$ and $B\mathbb{Z}/3\mathbb{Z}$, respectively, we find that $Y(\Gamma)$ has one elliptic point of order 2 and one of order 3. You can actually avoid orbifolds altogether if you’d prefer (which simply allow you to say the following more succinctly), just by noting that elliptic points of order 2 (resp. 3) correspond to conjugacy classes of order 2 (resp. 3) subgroups, which in an amalgamated product as in our case, is just the number of times $\mathbb{Z}/2\mathbb{Z}$ (resp. $\mathbb{Z}/3\mathbb{Z}$) appears in the amalgamated product. We plug these numbers into our genus formula, $g=1+\frac{7}{12}-\frac{1}{4}-\frac{1}{3}-\frac{\varepsilon_{\infty}}{2}=1-\frac{\varepsilon_{\infty}}{2}$. Since the genus is an integer, there must be an even number of cusps.
Now we apply theorem 2 of the linked paper, and look at the number of cusps. If there are two (the case of 4 or 6 are even easier), they must have indices $(1,6)$, $(2,5)$, or $(3,4)$, and in any case, 7 does not divide the order of $\operatorname{SL}_2(N)$ for $N=6,10$ or $12$, so that $\Gamma(N)$ has index not divisible by 7, and thus cannot be contained in $\Gamma$, a contradiction.
As a side note: using Euler characteristic of groups, one can show that any congruence subgroup of index 7 is isomorphic to $\mathbb{Z}/2\mathbb{Z}^{*3} *\mathbb{Z}/3\mathbb{Z}$, which shows there are 3 order 2 elliptic points and 1 order 3 elliptic point. For a similar reason to above, using theorem 2 of Wohlfahrt, there must be only one cusp, of index 7, and the genus of $X(\Gamma)$ must be zero.
I think this is a particularly simple construction and proof of a non-congruence subgroup, using only elementary covering space theory and knowledge about the genus of a modular surface. The question was already thoroughly answered in this thread before, but I find this way a bit easier than the answers already presented, and I’m hoping that somebody else benefits from seeing it.
