When does the double coset representative for a congruence subgroup contain a $\text{SL}_2(\mathbb{Z})$-conjugacy class?

In the paper p-adic L-functions and p-adic periods of modular forms, Greenberg/Stevens assert that if $$\sigma_l:=\begin{pmatrix}l&0\\0&1\end{pmatrix}$$ is the usual Hecke operator at $$l$$ double coset representative, and $$\Gamma$$ is the congruence group $$\Gamma_1(N)$$, then $$g\sigma_lg^{-1}$$ continues to lie in the same double coset class if $$l\equiv 1 \pmod{N}$$, for any $$g\in \text{SL}_2(\mathbb{Z})$$. I thought this would be straightforward to see via some sort of restricted row operations/Euclidean algorithm argument, but am struggling. What is the argument here, and what goes wrong when $$N$$ does not divide $$l-1$$?

Why the condition $$\ell = 1 \bmod N$$ is necessary. Suppose $$g \sigma_\ell g^{-1}$$ lies in the double coset $$\Gamma \sigma_\ell \Gamma$$. Then $$g \sigma_\ell g^{-1} = r \sigma_\ell s$$ for some $$r, s \in \Gamma_1(N)$$; but $$r \sigma_\ell s$$ is the product of three matrices which are upper-triangular mod $$N$$, so it must be upper-triangular mod $$N$$.
If we take $$g = \begin{pmatrix} 1 & 0 \\ 1 & 1 \end{pmatrix}$$, this amounts to requiring that $$\begin{pmatrix} \ell & 0 \\ \ell-1 & 1 \end{pmatrix}$$ be upper-triangular mod $$N$$, which is precisely the condition that $$\ell = 1\bmod N$$.
Why the condition is sufficient. One checks easily that if $$\ell \nmid N$$ then the double coset $$\Gamma_1(N) \sigma_\ell \Gamma_1(N)$$ contains every matrix of determinant $$\ell$$ which is congruent to $$\begin{pmatrix} * & * \\ 0 & 1 \end{pmatrix}$$ mod $$N$$. If $$\ell = 1 \bmod N$$, then $$g \sigma_\ell g^{-1}$$ satisfies these conditions, so it lies in the double coset $$\Gamma_1(N) \sigma_\ell \Gamma_1(N)$$.
Remark. For a general prime $$\ell$$ (even assuming $$\ell \nmid N$$), the double coset $$\Gamma_1(N) \sigma_\ell \Gamma_1(N) = \Gamma_1(N) \begin{pmatrix} \ell & 0 \\ 0 & 1 \end{pmatrix}\Gamma_1(N)$$ is not what is normally called $$T_\ell$$. It's sometimes known as $$T_\ell^*$$; and when $$\ell \nmid N$$ (as here), one has $$T_\ell^* = \langle \ell \rangle^{-1} T_\ell$$.
So there are really two assertions being blended together here, both of which need $$\ell = 1 \bmod N$$ to work: firstly, that the double coset $$T_\ell^*$$ is stable under conjugation; secondly, that it coincides with the double coset $$T_\ell$$.