If you do the computation for the "continued fraction generators", $\tau = \begin{pmatrix} 1 & 0\\0 & -1\end{pmatrix}$ and $\sigma = \begin{pmatrix} 1 & 1\\ 0 & 1\end{pmatrix},$ and write out the equation for $\begin{pmatrix} a & b\\ c & d\end{pmatrix}$ to conjugate $\sigma, \tau$ into integer matrices, you get the conditions that $a^2, b^2, c^2, d^2, ac, bd$ as necessary and sufficient. Which means that the normalizer consists of matrices of the form 
$\begin{pmatrix} x \sqrt{s} & y \sqrt{t}\\ z \sqrt{s} & w\sqrt{t}\end{pmatrix},$ where $x, y, z, w, s, t$ are integer and $xw\sqrt{st} - zy \sqrt{st} (xw - zy)\sqrt{st} = 1.$ From this it follows that $\sqrt{st} = 1,$ and thus $s=t=1.$