On the automorphism group of binary quadratic forms This question is a continuation of the following two questions:
Discriminants of indefinite integral binary quadratic forms admitting 3 or 6 torsion.
On certain solutions of a quadratic form equation
Let $f$ be a binary quadratic form with integer coefficients and non-zero discriminant. For $T = \begin{pmatrix}t_1 & t_2 \\ t_3 & t_4 \end{pmatrix}  \in \operatorname{GL}_2(\mathbb{R})$, define $f_T(x,y) = f(t_1 x + t_2 y, t_3 x + t_4 y)$. Put
$$\displaystyle \operatorname{Aut}(f) = \{T \in \operatorname{GL}_2(\mathbb{Z}) : f_T = f\}.$$
When $f$ is positive definite, then $\# \operatorname{Aut}(f)$ is easy to determine. In particular, if $f(x,y)$ is reduced, so that it is written as
$$\displaystyle f(x,y) = ax^2 + bxy + cy^2$$
with $|b| \leq a \leq c$ with $ac > 0$, then $\# \operatorname{Aut}(f) = 12$ when $a = b = c = 1$, $\# \operatorname{Aut}(f) = 8$ when $a = c = 1, b = 0$, $\# \operatorname{Aut}(f) = 4$ when $a = \pm b$, and $\#\operatorname{Aut}(f) = 2$ otherwise. When $f$ is reducible, the situation is also easy to account for. The most difficult case is when $f$ is indefinite and irreducible. 
One part of the problem is easy. If we consider only positive determinant elements, then they are given by
$$\displaystyle \begin{pmatrix} \dfrac{s+bt}{2} & & -ct \\ \\ at & & \dfrac{s - bt}{2} \end{pmatrix},$$
where $(s,t)$ is a solution to the Pell equation $x^2 - (b^2 - 4ac)y^2 = 4$ (note that this equation is always soluble since $(2,0)$ is a solution).
The problem is the elements of negative determinant. I can show (see the second question linked above) that such elements take the form
$$\displaystyle \begin{pmatrix} m & (bm + cn)/a \\ n & - m\end{pmatrix},$$
where $(m,n)$ is a solution to $am^2 + bmn + cn^2 = a$, which is again always soluble. But there is no guarantee that $(bm+cn)/a$ is an integer. This is guaranteed if $n$ is co-prime to $a$, since then looking at the equation above we see that $a | n(bm+cn)$, whence $a | bm + cn$. There are however examples where this doesn't happen; I believe $5x^2 + 3xy - 11y^2$ is such an example. All of the solutions to $5x^2 + 3xy - 11y^2 = 5$ have $5 | y$, and since $5 \nmid 3$ and $\gcd(m,n) = 1$ by necessity, it is impossible for $3m-11n$ to be divisible by 5 and thus no way for the above matrix to be integral. 
However, such examples seem extremely rare. I don't have a lot of computational evidence except to say that there is no such example with $D = b^2 - 4ac < 100$.
The question is, when is the set
$$\displaystyle G^-(f) = \{T \in \operatorname{Aut}(f) : \det T = -1\}$$
non-empty? There are several sufficient conditions for this to be non-empty: if $f$ is equivalent to a diagonal form, if $f$ is equivalent to a form such that $a | b$, , if $f$ is equivalent to a symmetric form, or $f$ is equivalent to a form such that there exists a solution $(m,n)$ to $am^2 + bmn + cn^2 = a$ with $\gcd(n,a) = 1$. Moreover, if the class number is one, then all forms with discriminant $d$ are equivalent and therefore must be equivalent to either $x^2 - (d/4)y^2$ if $d$ is even or $x^2 + xy - (d-1)y^2/4$ if $d$ is odd. Both cases have non-empty $G^-$. 
A related question is, suppose that $f_1(x,y) = a_1 x^2 - b_1 y^2$ and $f_2(x,y) = a_2 x^2 - b_2 y^2$ are two non-proportional binary quadratic forms such that $a_1 b_1 = a_2 b_2 > 0$. Then when can $f_1, f_2$ be $\operatorname{GL}_2(\mathbb{Z})$-equivalent?
 A: Here is a necessary and sufficient condition for $G^{-}(f)$ to be non-empty, taken from [1, Exercise 6.21]:


Let $S = \begin{pmatrix} a & b/2 \\ b/2 & c \end{pmatrix}$ with $a,b$ and $c \in \mathbb{Z}$. Then the following are equivalent:

$(1)$ There exists $A \in \text{GL}_2(\mathbb{Z})$ with $\det(A)= -1$ such that $^tASA = S$.
$(2)$ The matrix $S$ is $\text{SL}_2(\mathbb{Z})$-equivalent to $\begin{pmatrix} a & -b/2 \\ -b/2 & c \end{pmatrix}$.
$(3)$ There exists $S' = \begin{pmatrix} a' & b'/2 \\ b'/2 & c' \end{pmatrix}$ with $a', b'$ and $c' \in \mathbb{Z}$ which is $\text{SL}_2(\mathbb{Z})$-equivalent to $S$ such that $b'$ is divisible by $a'$. (Here we may take  $b' = a'$ or $b' = 0$.)

Hint: If $\,^tASA = S$ with $\det(A)= -1$, then show that $\text{Tr}(A) = 0$ and $A^2 = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix}$.
Then reduce the case when $A$ is upper triangular.
If the above conditions hold, the $\text{SL}_2(\mathbb{Z})$-class to which such an $S$ belongs is called an ambig class (sic). This is also what John Robertson calls an ambiguous class in [2]. From this source, we get


Let $D \in \mathbb{Z}$ which is not a square and such that $D \equiv 0$ or $D \equiv 1 \,(\text{mod } 4)$. Then the number of ambiguous classes is the order of the genus group of $D$, that is, the order of $H^{+}(D) \otimes \mathbb{Z}/2\mathbb{Z}$ where $H^{+}(D)$ is the strict class group of $D$.



[1] T. Ibukiyama, M. Kaneko, "Quadratic Forms and Ideal Theory of Quadratic Fields" in Bernoulli Numbers and Zeta Functions, pp 75-93, 2014.
[2] J. Robertson, "Computing in Quadratic Orders", 2009.
A: This may or may not be what you want... my sense is that the conditions you describe as sufficient are also necessary, in this case the bit about equivalent to a form $\langle a,b,c \rangle$ such that $a | b.$ This is from Dickson's Introduction to the Theory of Numbers, 1929. Well, you can judge for yourself.

