Let $N \ge 1, k \ge 2$ be integers, and $M_k(\Gamma_1(N))$ the space of weight k modular forms of level $\Gamma_1(N)$. Let $\mathbb{T}$ be the $\mathbb{Z}$-subalgebra of $\operatorname{End} M_k(\Gamma_1(N))$ generated by the diamond operators $\langle d \rangle$ and the Hecke operators $T_\ell$ for $\ell \nmid N$. (This is sometimes called the "anemic Hecke algebra" because we've omitted the Hecke operators at primes dividing $N$).

If we make $\mathbb{T}$ still more anemic, by omitting the $T_\ell$'s for some finite subset $\Sigma$ of the primes $\ell \nmid N$, is the resulting subalgebra $\mathbb{T}^{\Sigma}$ the whole of $\mathbb{T}$?

(This is certainly true after tensoring with $\mathbb{Q}$ by the strong multiplicity one theorem, and I managed to convince myself it should be true integrally by a complicated argument using Galois representations and Chebotarev density; but is there a slick proof that doesn't use such heavy machinery?)