This question is a continuation of the following two questions:

http://mathoverflow.net/questions/133445/discriminants-of-indefinite-integral-binary-quadratic-forms-admitting-3-or-6-tor

http://mathoverflow.net/questions/245094/on-certain-solutions-of-a-quadratic-form-equation

Let $f$ be a binary quadratic form with integer coefficients and non-zero discriminant. For $T = \begin{pmatrix}t_1 & t_2 \\ t_3 & t_4 \end{pmatrix}  \in \operatorname{GL}_2(\mathbb{R})$, define $f_T(x,y) = f(t_1 x + t_2 y, t_3 x + t_4 y)$. Put

$$\displaystyle \operatorname{Aut}(f) = \{T \in \operatorname{GL}_2(\mathbb{Z}) : f_T = f\}.$$

When $f$ is positive definite, then $\# \operatorname{Aut}(f)$ is easy to determine. In particular, if $f(x,y)$ is reduced, so that it is written as

$$\displaystyle f(x,y) = ax^2 + bxy + cy^2$$

with $|b| \leq a \leq c$ with $ac > 0$, then $\# \operatorname{Aut}(f) = 12$ when $a = b = c = 1$, $\# \operatorname{Aut}(f) = 8$ when $a = c = 1, b = 0$, $\# \operatorname{Aut}(f) = 4$ when $a = \pm b$, and $\#\operatorname{Aut}(f) = 2$ otherwise. When $f$ is reducible, the situation is also easy to account for. The most difficult case is when $f$ is indefinite and irreducible. 

One part of the problem is easy. If we consider only positive determinant elements, then they are given by

$$\displaystyle \begin{pmatrix} \dfrac{s+bt}{2} & & -ct \\ \\ at & & \dfrac{s - bt}{2} \end{pmatrix},$$

where $(s,t)$ is a solution to the Pell equation $x^2 - (b^2 - 4ac)y^2 = 4$ (note that this equation is always soluble since $(2,0)$ is a solution).

The problem is the elements of negative determinant. I can show (see the second question linked above) that such elements take the form

$$\displaystyle \begin{pmatrix} m & (bm + cn)/a \\ n & - m\end{pmatrix},$$

where $(m,n)$ is a solution to $am^2 + bmn + cn^2 = a$, which is again always soluble. But there is no guarantee that $(bm+cn)/a$ is an integer. This is guaranteed if $n$ is co-prime to $a$, since then looking at the equation above we see that $a | n(bm+cn)$, whence $a | bm + cn$. There are however examples where this doesn't happen; I believe $5x^2 + 3xy - 11y^2$ is such an example. All of the solutions to $5x^2 + 3xy - 11y^2 = 5$ have $5 | y$, and since $5 \nmid 3$ and $\gcd(m,n) = 1$ by necessity, it is impossible for $3m-11n$ to be divisible by 5 and thus no way for the above matrix to be integral. 

However, such examples seem extremely rare. I don't have a lot of computational evidence except to say that there is no such example with $D = b^2 - 4ac < 100$.

The question is, when is the set

$$\displaystyle G^-(f) = \{T \in \operatorname{Aut}(f) : \det T = -1\}$$

non-empty? There are several sufficient conditions for this to be non-empty: if $f$ is equivalent to a diagonal form, if $f$ is equivalent to a form such that $a | b$, or $f$ is equivalent to a form such that there exists a solution $(m,n)$ to $am^2 + bmn + cn^2 = a$ with $\gcd(n,a) = 1$. Moreover, if the class number is one, then all forms with discriminant $d$ are equivalent and therefore must be equivalent to either $x^2 - (d/4)y^2$ if $d$ is even or $x^2 + xy - (d-1)y^2/4$ if $d$ is odd. Both cases have non-empty $G^-$. 

A related question is, suppose that $f_1(x,y) = a_1 x^2 - b_1 y^2$ and $f_2(x,y) = a_2 x^2 - b_2 y^2$ are two non-proportional binary quadratic forms such that $a_1 b_1 = a_2 b_2 > 0$. Then when can $f_1, f_2$ be $\operatorname{GL}_2(\mathbb{Z})$-equivalent?