I'm looking for the asymptotic order of growth of the number of points in algebraic groups, such as $\mathrm{SL}_n(\mathbb{Z})$, of height/norm at most $X$, i.e. all entries are at most $X$ in absolute value. (One can ask the same question for other equivalent norms, such as the spectral $L^2$-norm, without changing the behavior very much; but note that I do NOT want to order matrices by the length of a word in a predetermined generating set, as Rivin does to skirt this question.)
For $\mathrm{SL}_2(\mathbb{Z})$, the rows must be successive terms of a Farey sequence of order at most $X$ (up to permutations and sign changes), giving $\asymp X^2$ matrices. What about higher $\mathrm{SL}_n$ (or other groups such as $\mathrm{Sp}_{2n}(\mathbb{Z})$ or the split orthogonal group)? Are there well-known nontrivial lower and upper bounds?