Write $\mathbb{Z}^{a\times b}$ for the $a\times b$ integer matrices. Let $n\geq 3$ and $Q\in\mathbb{Z}^{n\times n}$. Let $G=O(Q)$ be the orthogonal group of $Q$. For $X_0\in \mathbb{Z}^{2\times n}$, set $$ R'(T,Q,X_0) = \{ X\in X_0G(\mathbb{Z}) : |X_{ij}|\leq T\}. $$

**Dubious heuristic:** If $Q$ is indefinite then
$$
\#R'(T,Q,X_0) \ll_{Q,\epsilon}
T^{\max\{0,2n-6\}+\epsilon}
\tag{$\star$}
$$
for all $\epsilon>0$, with an implicit constant depending only on $Q$ and $\epsilon$.

**Question:** For which $Q$ and which $X_0$ is this true? How about simple (?) examples like $Q=\operatorname{diag}(1,1,-1)$ or $Q=\operatorname{diag}(1,1,-1,-1)$?

*Update 2018/4/11:* I wonder if the work of Gorodnik and Nevo allows one to prove upper and lower bounds of the form $R'(T,Q,X_0)\asymp_Q T^{\text{something}}$ for all $X_0$ with entries $\ll T^{\delta}$, for some small $\delta$. In particular this would answer the question for all small $X_0$. I will try to look into this further.

*Update 2018/4/11:* As he explains in his answer below, Emilio Lauret gave in his PhD thesis a general strategy to answer questions of this type. This uses the result of Gorodnik and Nevo mentioend above. There is nontrivial work still to be done to implement this strategy in any particular case, more specifically one must compute the constant $\delta$ from page 45, line -5 of his thesis. (See his answer for a link). Unless and until someone comes up with an ever better approach to the problem, I accept his answer!

## Background

Let $B\in\mathbb{Z}^{2\times 2}$ be nonsingular and symmetric. I am interested in the number of solutions $$ R(T,Q,B) = \{ X\in \mathbb{Z}^{2\times n} : XQX^T =B, \, |X_{ij}|\leq T\} $$ with height up to $T$, when $Q$ is indefinite. A naive guess based on the circle method would be that $$ \#R(T,Q,B) \ll_Q 1+T^{2n-6} \tag{$\ast$} $$ with an implicit constant depending at most on $Q$. The set $R(T,Q,B)$ is the union of finitely many $G(\mathbb{Z})$-orbits $R'(T,Q,X_0)$, so the conjecture above is a weak version of this.

Note that ($\ast$) is true for large $n$. For instance when $Q$ is indefinite and $n\geq 12$ this (and much more) follows from the asymptotic result of Brandes. (The condition $n\geq 12$ follows from the comment at the end of section 1 in the subsequent paper.) On the other hand ($\ast$) is certainly false for small $n$, though I don't know of a good example in the indefinite case.

If $B$ is positive definite and $Q$ has signature $(p,q)$ with $p\geq q$ then it is a special case of Theorem 1.4 of Eskin, Mozes and Shah that $$ \# R(T,Q,B) \sim C T^{\min\{2,q\}(n-1-\min\{2,q\})} \tag{$\dagger$} $$ for some $C= C(Q,B)$, unless a certain volume is infinite. In the latter case, the comments after formula (1.4) of Duke, Rudnick and Sarnak suggest that the right-hand side of ($\dagger$) just changes by a factor of $\log T$. If this is correct then ($\star$) can be false in the case when $(p,q)=(2,1)$ and $B=X_0Q X_0^T$ is positive definite, since $\# R(T,Q,B)\gg T$ for all $T\gg_{Q,B} 1$.

Perhaps a proof or disproof of ($\star$) in other cases is implicit in the work cited in the previous paragraph, but I have not been able to verify this.