Added later: One key word you might pursue is Patterson-Sullivan theory, which among other things equates the Hausdorff dimension of the limit set of a geometrically finite group action on hyperbolic space and the so-called "exponent of convergence" for that group action.
Original answer: Here is an answer to the question about known examples of Fuchsian subgroups of $SL(2,\mathbb{Z})$ whose limit set is a Cantor set, using a simple basic construction.
The action of $SL(2,\mathbb{Z})$ preserves the Farey tiling by ideal triangles. Dual to the Farey tiling is a trivalent tree $T$ which is invariant under the action. Elements $M \in SL(2,\mathbb{Z})$ with $|Trace(M)| \ge 2$ have an invariant line $L$ (meaning a bi-infinite edge path) in this tree. If $|Trace(M)| =2$ then $M$ acts "parabolically" and $L$ is a "quasi-horocycle" in the hyperbolic plane, having both ends converging to the same rational point at infinity. And if $|Trace(M)| > 2$ then $M$ is "loxodromic" and $L$ is a "quasi-geodesic" in the hyperbolic plane, having ends converging to distinct quadratic irrationalities (i.e. numbers with periodic continued fraction expansions).
Given two loxodromic transformations $M_1,M_2$ with distinct invariant lines $L_1,L_2$, the intersection $L_1 \cap L_2$ must be a finite segment $\alpha$, or a point, or the empty set. If the intersection is a fintie segment $\alpha$, and if for each $i$ there exists a subsegment $D_i \subset L_i$ which is a fundamental domain for the action of $M_i$ whose interior contain $\alpha$, then $M_1,M_2$ freely generate a rank $2$ Fuchsian group (if this was not already true for $M_1,M_2$, it will become true after replacing $M_1,M_2$ by sufficiently high powers). All the groups obtained from this construction are "convex cocompact" Fuchsian groups, meaning that they act cocompactly on the convex hull of their limit set. Also, if the intersection $L_1 \cap L_2$ is a point or the empty eset then $M_1,M_2$ automatically generate a convex cocompact Fuchsian group.
From convex cocompactness of a Fuchsian group which is free of rank $2$, it automatically follows that the limit set is homeomorphic to the end space aka Gromov boundary of the rank $2$ free group, which is a Cantor set.