Consider $\tau$ in the upper half plane such that $\tau$ is fixed by some $\alpha \in GL_2^{+}(\mathbb{Q})$ (i.e. positive determinant). Now suppose that there is $\tau'$ in the upper half plane such that $j(g \tau) = j(g \tau')$ for all $g \in GL_2^{+}(\mathbb{Q})$, where $j$ is the modular $j$-function. Then does $\tau = \tau'$?

Consider $\tau$ and $\tau'$ in the upper half plane such that $j(g \tau) = j(g \tau')$ for all $g \in GL_2^{+}(\mathbb{Q})$, where $j$ is the modular $j$-function and $GL_2^{+}(\mathbb{Q})$ acts as Mobius transformations. Then does $\tau = \tau'$?