Consider $\tau$ and $\tau'$ in the upper half plane such that $j(g \tau) = j(g \tau')$ for all $g \in GL_2^{+}(\mathbb{Q})$, where $j$ is the modular $j$function and $GL_2^{+}(\mathbb{Q})$ acts as Mobius transformations. Then does $\tau = \tau'$?
Unless you are proposing some action of $GL_2(\mathbb Q)^{+}$ on the upperhalf plane other than the obvious one, every value of $\tau$ is fixed by every scalar. In $PGL_2(\mathbb Q)^+$, the fixed points are CM points. But regardless, this is true for every value of $\tau$. Multiply both $\tau$ and $\tau'$ by $n$ using the matrix: $\left(\begin{array}{cc} n & 0 \\ 0 & 1 \end{array}\right)$ so that their imaginary parts are at least $1$. In this region of the upperhalfplane, $j(\tau)=j(\tau')$ implies $\tau'=\tau+n$ from some integer $n$. (This is obvious from the diagram of fundamental domains of the modular group.) Therefore their imaginary parts are the same. Then use the matrix: $\left(\begin{array}{cc} 0 & 1 \\ 1 & 0 \end{array}\right)$ to send $\tau$ to $1/\tau$, sending the horizontal line of fixed imaginary part to a semicircle. By the same logic as before, their imaginary parts must be equal, meaning that $\tau'=\tau$ or $\tau'=\bar{\tau}$. We can pull that equation back along the matrices that we used, so we know the original $\tau'$ is either $\tau$, in which case we are done, or $\bar{\tau}$ (and $Re(\tau)\neq 0$.) In the second case, send $\tau$ to $\tau+1$ with the matrix: $\left(\begin{array}{cc} 1 & 1 \\ 0 & 1 \end{array}\right)$ and now neither equation holds, which is a contradiction by the preceding argument. So $\tau=\tau'$. 

