I Just saw Scott's answer as I was preparing to post this addition. There is a bit of overlap.

Added, 19 November.

After receiving the nice answers from Jeff and Antoine, I realized that I should sharpen the question somewhat more. I like Neil's formulation, but I think I am asking something much more naive. Allow me to start by providing more background for mathematicians whose knowledge is as hazy as mine. As mentioned by Jeff and Antoine, the Hamiltonian for a system of $N$ electrons moving around a nucleus of charge $Z$ looks like

$$H=-\frac{\hbar^2}{2m}\sum_{i=1}^N \Delta_i-\sum_{i=1}^N \frac{Ze^2}{r_i}+ \sum \frac{e^2}{r_{ij}}.$$

In principle, one would like to understand the structure of discrete spectrum solutions to the eigenvalue equation $$H\psi=E\psi.$$ One characterizes solutions using labels called 'quantum numbers'. In general, there is an * objective label* often denoted $l$, the angular momentum quantum number. This comes from the fact that there is an $SO(3)\times SU(2)$ symmetry, breaking up solutions into the irreducible representation $V_l\otimes S$, mentioned earlier. All the vectors in a given irreducible representation must have the same energy level $E$, because a basis for the space can be obtained from a highest weight vector by applying elements of $LieSO(3)$. Any given $V_l\otimes S$ is called a subshell or an orbital (I'm a bit unclear about this because the term 'orbital' is also used for the individual eigenfunctions) and is described using the (historical) labels

$$V_0\otimes S \leftrightarrow s$$ $$V_1\otimes S \leftrightarrow p$$ $$V_2\otimes S \leftrightarrow d$$ $$V_3\otimes S \leftrightarrow f$$

The * principal quantum number* is determined as follows. We order all the orbitals by energy levels, and the $n-$th time $V_0 \times S$ occurs, we call that subspace $ns$. However, the $n$-th time that $V_1\otimes S$ occurs, we call it $(n+1)p$. In general, the $n$-th time $V_l\otimes S$ occurs, the label is $(n+l+1)$(whatever letter $l$ corresponds to). For example, the orbital $3d$ is the first occurrence of the representation $V_2\otimes S$. When there is just one electron in a $1/r$ potential, these $n$ really label the order of the energy levels, and $d$ really occurs the first time in the third energy level. This is the reason for the shift in labelling, even though this correspondence with energy levels fails for multi-electron systems. In fact, in neutral atoms with $N=Z$, the energy levels are ordered like $$1s<2s<2p<3s<3p<3d<4s<4p<5s<4d\ldots$$ I hope it is clear then that the orbitals (or the subshells) are completely well-defined and have a labelling scheme that is a bit odd, but makes sense when the obvious translation is combined with history. So the real question is

What determines a shell?

Some energy-non-decreasing consecutive sequence of subshells is a shell. The shells then determine the rows of the periodic table (if we ignore the added complication that $Z$ is also increasing as we move right and down). Now, I was tempted to conclude that the standard convention was a bit arbitrary. After all, the chemical similarity of the columns could possibly be explained simply by the fact that

they have the same representation of $SO(3)\times SU(2)$ at the uppermost energy level, and the same number of electrons in this representation

without any reference to an outermost shell. This can be easily seen by the arrangement into blocks shown here . And then, unlike the $N=1$ case, the energy levels in one shell are not even the same, just rather close to each other.

Unfortunately, it is clear that

the grouping into rows represent real phenomena.

This comes out very clearly, for example, in the graph of ionization energies, with the noticeable peaks at the end of the rows immediately followed by precipitous drops. So I believe a bit of thought reduces a good deal of my original question to two parts:

1. Is there some reason for a big gap in ionization energy when one moves to an $s$-orbital?

2. Can one show that two orbitals of the same type are necessarily separated by a huge energy gap, so that it is highly unlikely for them to occur in the same shell?

1 and 2 together make it natural that the dimensions we see in each shell will be of the form

2, 2+6, 2+6+10, 2+6+10+14, etc,

even in the general case. Of course, this doesn't say anything about how many times each combination is likely to occur.

In any case, I hope there is a mathematical answer to these two questions that doesn't involve a full-blown programme in hard analysis.

At the more speculative end, one might analyze a family of operators like

$$H_{\epsilon}=-\frac{\hbar^2}{2m}\sum_{i=1}^N \Delta_i-\sum_{i=1}^N\frac{Ze^2}{r_i}+\epsilon \sum \frac{e^2}{r_{ij}}.$$ Is there a way to see the numbers we see occurring via the spectral flow of this family as we go from $\epsilon =0 $ to $\epsilon=1$? But maybe this is just as difficult as giving a full account of the structures using analysis.

By the way, I certainly wouldn't like to complain, but it is a bit puzzling to me why some people regard this question as inappropriate for the site. Since I don't keep too well in touch with cultural trends in the mathematical world, maybe I am unaware of how much things have changed since I was a Ph.D. student. In those days, a programme like

Prove the stability of matter, based only on the Schroedinger equation

was regarded as an example of an important mathematical problem motivated by atomic structure, tackled by people like Fefferman and Lieb. The questions I ask here are hopefully much easier, but still research-level mathematics to my mind.

This question prompts me to ask something more specific about the periodic table. As far as I know, the main significance of the periodic table is that

The elements in the same column have similar chemical properties.

For example, the noble gases at the far right are all pretty stable on their own. The explanation for this is that they have (in the neutral state) a full outermost shell. Now my question is

Is there an explanation, at some reasonable level of mathematical rigor, of when a new shell starts?

That is, where do the lengths of the periods

2, 8, 8, 18, 18, 32, 32

come from? I confess I've been puzzled by this ever since my university physics course.

Allow me to pinpoint my confusion a bit more. I understand that there are some numbers that are important in atomic structure, and these are 2, 6, 10, 14, and so on. This is because of the occurrence of the representations

$$V_l\otimes S$$

inside the Hilbert space for a single particle moving in a central potential. These are the orbitals one hears about in physical chemistry courses. Here, the $V_l$ are the representations of $SO(3)$ of odd-dimension $2l+l$, while $S$ is the standard two-dim representation of $SU(2)$. Since all the states in a single orbital have the same energy, it is natural that the breaks will occur after some collection of orbitals are all filled.

Thus, for a hydrogen atom, the successive shells have dimensions

$2n^2$

for $n=1, 2, 3, \ldots$ because the representations $V_l\otimes S$ for $l=1, 2,\ldots, n-1$ occur each with multiplicity one inside the $n$-th shell.

So if the periods in the table were of lengths

2, 8, 18, 32,

I would have vaguely assumed that the pattern of shells even in general looks like that of the hydrogen atom. But of course, among the known elements, each period length that occurs in the hydrogen atom is repeated twice, except for the first one. So a more precise question is

Is there a reasonable mathematical explanation of this `multiplicity two' of the periods?

The little I recall of discussions in standard textbooks were quite unclear. There are various rules by the name of Hund's rule, the Aufbau principle, and so on, but I couldn't gather from any of them

*Where the breaks should occur. *

What I do see is that periods end when the orbitals 1p, 2p, 3p, etc. get filled following the Aufbau principle. (Here, p is the chemist's label for $V_1\otimes S$.) So perhaps another version of the question is

Is there some reason that the p-orbitals mark the end of the periods?

To be honest, I've never understood the Aufbau principle either, because I don't know the rationale behind the principal quantum number for the larger atoms. That is, the number 4 in the orbital 4p refers to the 4-th energy level in the case of the hydrogen atom. But for larger nuclei, the $n $ in orbital '$np$' does not refer to the energy level. (This discrepancy is in fact implied by the Aufbau principle.) So what is the significance of the $n$ in general that enables them to play some role in a physical principle?

I realize this question is becoming incoherent already. Nevertheless, I would very much appreciate clarification on any sensible version of it at a level of mathematical rigor of your choice. (I am not asking for any axiomatics.) Pointers to an accessible reference would be equally welcome.

As with the earlier question, a word of explanation is in order on the decision to post this on Math Overflow. I will draw upon an analogy I read long ago in an article of George Mackey's that went something like this: Say my mother tongue is Korean. If I would really like to use English fluently, it is probably best eventually to learn from native speakers of English. On the other hand, if I would like a good translation into Korean of English literature, it is better to consult an educated Korean who knows a lot about English. Of course answers from real physicists will be very gratefully received, especially if they bear in mind that the query comes from someone who struggles against a serious language handicap.