Upper bounds for lattice points in orbits, and representations of binary quadratic forms

Question

Write $\mathbb{Z}^{a\times b}$ for the $a\times b$ integer matrices. Let $n\geq 3$ and $Q\in\mathbb{Z}^{n\times n}$. Let $G=O(Q)$ be the orthogonal group of $Q$. For $X_0\in \mathbb{Z}^{2\times n}$, set $$ R'(T,Q,X_0) = \{ X\in X_0G(\mathbb{Z}) : |X_{ij}|\leq T\}. $$

Dubious heuristic: If $Q$ is indefinite then $$ \#R'(T,Q,X_0) \ll_{Q,\epsilon} T^{\max\{0,2n-6\}+\epsilon} \tag{$\star$} $$ for all $\epsilon>0$, with an implicit constant depending only on $Q$ and $\epsilon$.

Question: For which $Q$ and which $X_0$ is this true? How about simple (?) examples like $Q=\operatorname{diag}(1,1,-1)$ or $Q=\operatorname{diag}(1,1,-1,-1)$?

Update 2018/4/11: I wonder if the work of Gorodnik and Nevo allows one to prove upper and lower bounds of the form $R'(T,Q,X_0)\asymp_Q T^{\text{something}}$ for all $X_0$ with entries $\ll T^{\delta}$, for some small $\delta$. In particular this would answer the question for all small $X_0$. I will try to look into this further.

Update 2018/4/11: As he explains in his answer below, Emilio Lauret gave in his PhD thesis a general strategy to answer questions of this type. This uses the result of Gorodnik and Nevo mentioend above. There is nontrivial work still to be done to implement this strategy in any particular case, more specifically one must compute the constant $\delta$ from page 45, line -5 of his thesis. (See his answer for a link). Unless and until someone comes up with an ever better approach to the problem, I accept his answer!

---

Background

Let $B\in\mathbb{Z}^{2\times 2}$ be nonsingular and symmetric. I am interested in the number of solutions $$ R(T,Q,B) = \{ X\in \mathbb{Z}^{2\times n} : XQX^T =B, \, |X_{ij}|\leq T\} $$ with height up to $T$, when $Q$ is indefinite. A naive guess based on the circle method would be that $$ \#R(T,Q,B) \ll_Q 1+T^{2n-6} \tag{$\ast$} $$ with an implicit constant depending at most on $Q$. The set $R(T,Q,B)$ is the union of finitely many $G(\mathbb{Z})$-orbits $R'(T,Q,X_0)$, so the conjecture above is a weak version of this.

Note that ($\ast$) is true for large $n$. For instance when $Q$ is indefinite and $n\geq 12$ this (and much more) follows from the asymptotic result of Brandes. (The condition $n\geq 12$ follows from the comment at the end of section 1 in the subsequent paper.) On the other hand ($\ast$) is certainly false for small $n$, though I don't know of a good example in the indefinite case.

If $B$ is positive definite and $Q$ has signature $(p,q)$ with $p\geq q$ then it is a special case of Theorem 1.4 of Eskin, Mozes and Shah that $$ \# R(T,Q,B) \sim C T^{\min\{2,q\}(n-1-\min\{2,q\})} \tag{$\dagger$} $$ for some $C= C(Q,B)$, unless a certain volume is infinite. In the latter case, the comments after formula (1.4) of Duke, Rudnick and Sarnak suggest that the right-hand side of ($\dagger$) just changes by a factor of $\log T$. If this is correct then ($\star$) can be false in the case when $(p,q)=(2,1)$ and $B=X_0Q X_0^T$ is positive definite, since $\# R(T,Q,B)\gg T$ for all $T\gg_{Q,B} 1$.

Perhaps a proof or disproof of ($\star$) in other cases is implicit in the work cited in the previous paragraph, but I have not been able to verify this.

Answer 1

This is not an answer, but a long comment providing related known results.

In the particular case $Q=I_{n,1}= diag(I_n,-1)=diag(1,\dots,1,-1)$ (called the Lorentzian case), Ratcliffe and Tschantz gave in 1 an asymptotic formula for the number of $x=(x_1,\dots,x_{n+1})^t\in \mathbb Z^{n+1}$ satisfying that $x^tI_{n,1}x=k$ ($k\in\mathbb Z$) and $\|x\|^2\leq t$, as $t\to\infty$. When $k$ is negative, they applied (in a clever way) the lattice point theorem by Lax and Phillips, obtaining a formula with an error term.

In my Ph.D. thesis (available at my web page, though it is written in Spanish) I extended the above formulas for more general indefinite quadratic or hermitian forms $Q$ of signature $(n,1)$ (see [2]). Furthermore, I also worked on counting certain solutions of $X\in \mathbb Z^{(p+q)\times q}$ satisfying $I_{p,q}[X]=X^*I_{p,q}X=-L$, where $L$ is a positive definite integral matrix by using a lattice point theorem by Gorodnik and Nevo [3] or [4]. Unfortunately, concerning the last case, it is not easy to compute explicitly the constants involved in the asymptotic formulas.

I hope similar technics as above may help to find upper bounds like those requested in the question.

1 Ratcliffe, John G.; Tschantz, Steven T., On the representation of integers by the Lorentzian quadratic form, J. Funct. Anal. 150, No. 2, 498-525 (1997). ZBL0883.11017.

[2] Lauret, Emilio A., An asymptotic formula for representations of integers by indefinite Hermitian forms, Proc. Am. Math. Soc. 142, No. 1, 1-14 (2014). ZBL1329.11029.

[3] Gorodnik, Alexander; Nevo, Amos, The ergodic theory of lattice subgroups, Annals of Mathematics Studies 172. Princeton, NJ: Princeton University Press (ISBN 978-0-691-14185-5/pbk; 978-0-691-14184-8/hbk). 136 p. (2009). ZBL1186.37004.

[4] Gorodnik, Alexander; Nevo, Amos, Counting lattice points, J. Reine Angew. Math. 663, 127-176 (2012). ZBL1248.37011.