Does there exist Fuchsian groups, which is not conjugated in $SL(2, \mathbb{R})$ to a subgroup of $SL(2, \mathbb{Z})$, but still contains a congruence subgroup?
How about the normalizer of $\Gamma_0(N)$ in $PSL_2(\mathbb{R})$ for $N > 1$, which is the subgroup generated by $\Gamma_0(N)$ and $\begin{pmatrix} 0 & 1 \\ N & 0 \end{pmatrix}$?
(I wouldn't call these "noncongruence subgroups" as you do in your title: that technical term is usually parsed as "(noncongruence) subgroup". This isn't a subgroup of $PSL_2(\mathbb{Z})$ at all, but it certainly contains a principal congruence subgroup; perhaps we should call it a "congruence nonsubgroup"?)

2$\begingroup$ This normalizer is indeed a discrete subgroup of $\mathrm{PSL}_2(\mathbf{R})$. For the property that no conjugate is contained in $\mathrm{PSL}_2(\mathbf{Z})$, I think one needs some assumption on $N$. This is true for example if $N$ is prime (this is Exercise 1.46 in Shimura's book Intro. to the arithm. theory of autom. functions). But in general I think there are counterexamples e.g. for $N=p^4$ with $p$ prime. $\endgroup$ – François Brunault Jan 27 '12 at 10:50

$\begingroup$ Okay, I recognize them now as the other maximal compact subgroup, which are useful when classifying supercuspidals. $\endgroup$ – Marc Palm Apr 6 '12 at 15:19