I suspect that I'm asking (familiar?) questions from deformation theory in a different language. But I'm an illiterate in deformation theory language; if my suspicion is correct I'd be grateful for an explanation of the connection between the two languages.

NOTATION

Fix a prime $\ell$, an $N>0$ prime to $\ell$, and an even integer $w$ in $[0,\ell-1)$.

Definition----An element $h$ of $\mathbb Z/\ell[\![x]\!]$ is in $M(w)$ if there is a modular form for $\Gamma_0(N)$, of weight congruent to $w\mod\ell-1$, whose expansion at infinity lies in $\mathbb Z[\![x]\!]$ and reduces mod $\ell$ to $h$. (It's easy to see that $M(w)$ is a $\mathbb Z/\ell$ vector-space).

Now for each $n$ prime to $\ell N$ there is a formal Hecke operator $T_n$, $\mathbb Z/\ell[\![x]\!]\to\mathbb Z/\ell[\![x]\!]$, depending on the "weight" $w$. For our fixed $w$, these $T_n$ span a Hecke algebra $HE(w)$; furthermore $HE(w)$ stabilizes $M(w)$.

Definition----An ideal of $HE(w)$ is modular if it is the annihilator of some non-zero $h$ in $M(w)$. It is maximal modular if it is maximal among such annihilators. (If $u$ is in $HE(w)$, then the annihilator of $u(h)$ contains the annihilator of $h$. So maximal modulars are prime. Also if $J=\operatorname{Ann}(h)$ is maximal modular, then since all transforms of $h$ under $HE(w)$ lie in a finite dimensional $\mathbb Z/\ell$ space, $HE(w)/J$ is a domain finite over $\mathbb Z/\ell$. So maximal modulars are maximal ideals).

Definition----If $J$ is maximal modular, $M(w,J)$ is the set of $h$ in $M(w)$ annihilated by some power of $J$.

QUESTIONS

1)---Is it true that there are only finitely many maximal $J$ in $HE(w)$? Is $M(w)$ the direct sum of the $M(w,J)$?

---Suppose now that $\ell=2$ so that $w=0$. Joel Bellaiche told me that when $N=1,3,5,7$ or $9$ and $F=x+x^9+x^{25}+...$, then $\operatorname{Ann}(F)$ acts locally nilpotently on $M(w)$. So $\operatorname{Ann}(F)$ is the only maximal modular in these cases, and one sees easily that $\operatorname{Ann}(F)=\operatorname{Ann}(r)$ for some $r$ "coming from a weight $4$ form for $\Gamma_0(N)$".

2)----Suppose $\ell=2$ so $w=0$. Does every maximal modular have the form $\operatorname{Ann}(h)$ for some $h$ coming from a form of weight 2 or 4? Is there some corresponding result for odd $\ell$?

---For a given maximal modular $J$ it seems to me that one can imitate the construction done by Nicolas and Serre in the case $\ell=2$, $N=1$, getting a "$J$-completed" version of $HE(w)$ acting faithfully on $M(w,J)$. In the Nicolas-Serre situation this $J$-completion is a two-variable power series ring over $\mathbb Z/2$.

3)----What is known about the $J$-completed algebra for other values of $\ell$, $N$ and $w$?

EDIT___I'm grateful to Kevin Ventullo for calling my attention to the preprint of Bellaiche and Khare for the case $N=1$. Though my definition of the $J$-completion of $HE$ and theirs differ in some ways, I imagine that they and I are talking about the same objects, and that when $N=1$, their conjecture that one always gets a 2-variable power series ring and their positive results supporting this conjecture apply to my question.

For $N>1$, I've only looked at the case $\ell=2$, $N$ an odd prime. Since $\ell=2$, $w=0$, and I'll omit the w in my notation, writing $M$, $M(J)$ and $HE$. Let $J$ in $\mathbb Z/2[\![x]\!]$ be $x+x^9+x^{25}+...$ as in question 1. When $J=\operatorname{Ann}(F)$ (which is the maximal ideal of $HE$ containing all the $T_p$, $p$ an odd prime other than N) it seems that my definition of the $J$-completion is "wrong". This is because of the presence of "oldforms" such as powers of $F$ and powers of $G=F(x^N)$ in $M(J)$. One should probably replace $M(J)$ by its quotient by the space spanned by these elements, and use the action of $HE$ on the quotient to define the $J$-completion. I'll leave this aside for now and assume that $J$ isn't $\operatorname{Ann}(F)$. Then $N$ is at least $11$. When $N=11$, there's only one other choice for $J$; $\operatorname{Ann}(t)$ where $t$ is the reduction of the weight $2$ newform for $\Gamma_0(11)$. This $J$ contains the elements $T_3 +I$, $T_5 +I$, and $T_7$.

I'll discuss this case, $N=11$ and $J=\operatorname{Ann}(t)$, for the rest of this edit. Though I have no proof, the computer convincingly suggests that the $J$-completion contains nilpotents, and is in fact a 2-variable power series ring over $\mathbb Z/2$, with an element of square 0 adjoined.

Let $M(\mathrm{odd},J)$ be the subspace of $M(J)$ consisting of $h$ in which all the exponents of $x$ are odd. It's easy to see that the $J$-completion of $HE$ doesn't change when the action on $M(J)$ is replaced by the action on $M(\mathrm{odd},J)$. Now $M(\mathrm{odd},J)$ is on the face of it a very complicated $HE$-module. However it has an $HE$-stable subspace $C(J)$ that is apparently easier to handle. (I'll define $C(J)$ shortly). Let $pr:\mathbb Z/2[\![x]\!]\to\mathbb Z/2[\![x]\!]$ be the map that removes from $h$ all terms of the form $x^{11k}$. It turns out that $pr$ is 1-1 on $C(J)$ so that studying the action of $HE$ on $C(J)$ is the same as studying the action on $pr(C(J))$. I've made computer calculations concerning this action, and will report the results on question 135902--"Higher level analogs of Nicolas-Serre theory". What the computer suggests is the following. Let $x$, $y$ and $z$ be the images of $T_3 +I$, $T_5 +I$ and $T_7$ in the $J$-completion of $HE$ acting on $pr(C(J))$. Then the map from the 3-variable power series ring to this $J$-completion is onto, and the kernel is a principal ideal generated by $f^2$, where $f=Z+X+Y+X^2+XY+Y^2+(X^2)Y+X(Y^2)+Y^3+$ higher degree stuff in $X$ and $Y$.

I've now found a proof that $M(\mathrm{odd}, J)=C(J)$. So the results of the last paragraph apply equally to the $J$-completion of $HE$ acting on $M(J)$. So if one believes the computer the situation is quite different from that of level 1.

$C$ and $M_0$ are defined in my question 138495--"Are these two subspaces of $\mathbb Z/2[\![x]\!]$ the same?" (When $N=11$ they are; this is essential to my arguments). I define $M_0$ to be the subspace of $M(\mathrm{odd})$ consisting of elements whose trace from $\mathbb Z/2(F,G)$ to $\mathbb Z/2(G)$ is zero. (Here $F$ is as above and $G=F(x^{11})$; $M$ is just the integral closure of $\mathbb Z/2[G]$ in $Z/2(F,G)$). $C=M_0$ is, like $M$ and $M(\mathrm{odd})$, stable under $HE$. Also it admits a direct sum decomposition with $J$ locally nilpotent on the first summand, which I call $C(J)$, and $\operatorname{Ann}(F)$ locally nilpotent on the second.

To show that $M(\mathrm{odd}, J)=C(J)$ is to show that every element of $M(\mathrm{odd},J)$ has trace $0$. The argument proceeds in several steps. (The first 2 generalize to arbitrary $N$).

1)__Suppose $m$ is odd and $T_{11}(F^m)$ is $\sum((c_i)(F^i))$. Then the trace of $F^m$ is $\sum((c_i)(G^i))$.

To see this let $K$ be an algebraic closure of $\mathbb Z/2$. For $\alpha$ in $K$ with $\alpha^{11}=1$, let $F_\alpha$ be $F((\alpha*(x^{\frac1{11}})))$. Then $G$ and the $F_\alpha$ are the conjugates of $G$ over $K(F)$. So the trace of $G^m$ from $\mathbb Z/2(F,G)$ to $\mathbb Z/2(F)$ is the sum of the $m$th power of $G$ and the $m$th powers of the $F_\alpha$. This sum is easily seen to be $T_{11}(F^m)=\sum((c_i)(F^i))$. We now use the fact that the irreducible equation between $F$ and $G$ is symmetric in $F$ and $G$ to derive 1).

2)__If $m$ is odd, $\operatorname{trace}(T_3(F^m))=T_3(\operatorname{trace}(F^m))$

For 1) shows that the left hand side is the result of replacing $F$ by $G$ in the element $T_{11}(T_3(F^m))$ of $\mathbb Z/2[F]$, while the right hand side is similarly obtained from $T_3(T_{11}(F^m))$.

3)__If $h$ is in $M(\mathrm{odd})$, then $\operatorname{trace}(T_3(h))=T_3(\operatorname{trace}(h))$

Since $M_0$ is stable under $HE$ (this uses the fact that $M_0=C$), it is stable under $T_3$, and the result holds for $h$ in $M_0$. In view of 2) it suffices to show that $M_0$ and the $F^m$ with $m$ odd span $M(\mathrm{odd})$. In view of 1) it's enough to show that $T_{11}$ maps the space spanned by the $F^m$ with $m$ odd ONTO itself. But this is immediate from the level 1 Nicolas-Serre theory.

The proof that each element $h$ of $M(\mathrm{odd},J)$ has trace 0 is now easy. $(T_3 +I)^q$ annihilates $h$ for some $q$, and we may take $q$ to be a large power of $2$. then $h=((T_3)^q)(h))$. So by 3), $\operatorname{trace}(h)=((T_3)^q)(\operatorname{trace}(h))$. But $T_3$ is locally nilpotent on the space spanned by the powers of $G$ giving the result.

It would be good to give a proof that what the computer makes clear is true is in fact the case. But $M(\mathrm{odd},J)$ is so much more complicated than the space spanned by the odd powers of $F$, that I'm not optimistic about an approach to this in the style of Nicolas and Serre.

EDIT (1/20/14)---SOME EMPIRICAL LEVEL 1 RESULTS

In level 1, when $\ell-1$ divides $12$, it isn't hard to show that each maximal modular $J$ in $HE(w)$ has residue-class-field equal to $\mathbb Z/\ell$. When $\ell$ is $2$ or $3$ (so that $w=0$), it's known that there's only one $J$, and that the image of $T_p$ in $\mathbb Z/\ell$ is $p+1$.

I've made computer calculations with $\ell = 5$, $7$ and $13$; they suggest the following striking results (which perhaps follow from the proof of Serre's modular forms conjecture?)

(a)--- When $\ell$ is $5$, for each $w$ in $2\mathbb Z/4\mathbb Z$ there are two $J$. When $w=0$, the image of $T_p$ in $HE(0)/J$ is $(p^3)+1$ for one of these, $(p^2)+p$ for the other. When $w=2$, the image of $T_p$ in $HE(2)/J$ is $p+1$ for one $J$, $(p^3)+(p^2)$ for the other.

(b)---When $\ell$ is $7$, for each $w$ in $2\mathbb Z/6\mathbb Z$ there are three $J$. When $w=0$, the images of $T_p$ are $(p^5)+1$, $(p^4)+p$, and $(p^3)+(p^2)$. When $w=2$, the images are $p+1$, $(p^4)+(p^3)$ and $(p^5)+(p^2)$. When $w=4$, they are $(p^3)+1$, $(p^2)+p$ and $(p^5)+(p^4)$.

(c)---When $\ell$ is $13$, for each $w$ in $2\mathbb Z/12\mathbb Z$ there are eight $J$. For six of these $J$ there are $a$ and $b$ in $\mathbb Z/12\mathbb Z$ with $a+b=w-1$, such that the image of $T_p$ in $HE(w)/J$ is $(p^a)+(p^b)$. But I don't yet understand the other two $J$. (I guess what I'm saying is that these six $J$ correspond to reducible Galois representations, and I'd like a representation-theoretic description of the other two $J$).

EDIT--I've found a lot of what I'm looking for in Bas Edixhoven's paper, "The weight in Serre's conjecture on modular forms", Invent. Math. 109, 563-594 (1992). In his discussion of his Theorem 3.4 he writes "The fact that eigenforms of weight at most $p+1$ give, up to twist, all systems of eigenvalues has been known to be true for a long time .... The first published proof ... for $p>5$, is due to Ash and Stevens".

This answers my question on the finiteness of the number of maximal modular ideals in any level. And in level 1, it shows that my last edit indeed gives all the maximal modulars when $\ell$ (that is to say $p$ in the notation of the last paragraph) is $5$, $7$ or $13$. (I also found an arXiv paper by Craig Citro and Alexandru Ghitza, "Enumerating Galois representations in Sage" , which lists, in level 1, the number of these ideals for $\ell$ up to 211).