The matrix $M = \left(\begin{smallmatrix} a & b \\ c & d \end{smallmatrix} \right)$ need not be in $\operatorname{GL}_2(\mathbb{Z})$. For example, take the quartic form

$$\displaystyle F(x,y) = x^4 + 3x^3 y + 5x^2 y^2 - 21xy^3 + 49y^4.$$

One checks that $F$ is fixed by the matrix given by $\frac{1}{\sqrt{7}} \left(\begin{smallmatrix} 0 & 7 \\ -1 & 0 \end{smallmatrix} \right)$ under substitution, and this implies that the roots of $F(x,1)$ are permuted by the Mobius transformation

$$\displaystyle x \mapsto \frac{7}{-x}.$$

If you assume a priori that $M = \left(\begin{smallmatrix} a & b \\ c & d \end{smallmatrix} \right) \in \operatorname{SL}_2(\mathbb{Z})$ and $F$ is irreducible, then $M$ must be an automorphism. Indeed, since the Galois group of $F(x,1)$ acts transitively on the roots of $F(x,1)$ since $F$ is assumed to be irreducible, it follows that you can apply Galois action on $\beta, \alpha$ to obtain an equation of the shape

$$\displaystyle \theta_1 = \frac{a \theta_2 + b}{c \theta_2 + d}$$

for any root $\theta_1$ of $F(x,1)$, so that all of the roots of $F(x,1)$ are related to another root by the same Mobius transformation (note that the Galois group of $F$ acts trivially on $a,b,c,d$, since we assumed that these are rational integers). Thus the Mobius transformation given by the matrix $M = \left(\begin{smallmatrix} a & b \\ c & d \end{smallmatrix} \right)$ permutes the roots of $F(x,1)$. Since $M \in \operatorname{SL}_2(\mathbb{Z})$, it is discriminant preserving and one can check that it fixes the leading coefficient of $F(x,y)$ as well, and since it permutes the roots and fixes the leading coefficient, it must be an automorphism.