Quantum mechanics outside $L^{2}$ spaces

Question

To this day, it is known that a satisfying mathematical formulation of quantum field theory is far from sight, even though some noninteracting theories can be described in rigorous mathematical language.

On the other hand, quantum mechanics is (at least to my knowledge) highly considered as a subject in which a rigorous mathematical treatment for a more mathematical audience is avaiable. There are plenty of books dedicated to introduce quantum mechanics from a mathematical perspective, and many others use quantum mechanics as an application to functional analysis and related concepts.

Ever since I got in touch with this subject from the first time, I noticed that most books (if not all books I know) provide discussions which are centered in $L^{2}$ spaces, which is an appropriate space to describe wave functions. Von Neumann's work on spectral theory provides the central tool to understand the principles of quantum mechanics from a mathematical solid point of view. However, I had the feeling that Dirac's approach to quantum mechanics was not usually being taken into account in those books; some books mention Dirac's notation and some of them mention the fact that some objects in his approach do not live in a Hilbert space, like position or momentum eigenvectors, but these discussions are usually brief and nothing else is said about it.

I think this feeling was reinforced after my this previous post of mine, where I ask why are Rigged Hilbert spaces are still some sort of research material rather than book material.

On the other hand, the more I get in touch with physics books about quantum mechanics, the more I see how the use of Dirac's approach is important to the theory. Depending on what you are studying, Dirac's approach becomes the standard way to do it. For instance, spin systems, path integrals and perturbation theory are usually described under Dirac's point of view in physics textbooks. In addition, I had to face expressions like: $$|\psi\rangle=\int dx\psi(x)|x\rangle \tag{1} \label{1}$$ countless times in physics books in order to understand some concept or theory, and the latter can only be fully understood using Rigged Hilbert spaces.

In summary, it seems to me that $L^{2}$ spaces provide a nice tool to study the basics of quantum mechanics from a mathematical point of view, but by any means it seems enough. Physicists don't always talk about wave functions. So, my question is: to what extent are $L^{2}$ spaces enough and to what extent can we say that quantum mechanics has a solid mathematical treatment? I believe the answer to this question might pass through one of the following points (but not necessarily): (1) quantum mechanics is not fully understood from a mathematical point of view and my premises are false and (2) there exists a way of fully translating every concept or system studied by physicists to some $L^{2}$ (maybe some isomorphism theorem?) in which case $L^{2}$ spaces are enough.

Answer 1

The key term in my opinion is direct integral.

A good framework for non-normalizable quantum states that the comments have not mentioned is that of the direct integral Hilbert space. Despite going all the way back to Von Neumann, these concepts are not very well-known and deserve more exposure.

It is not too difficult, by breaking your Hilbert space into cyclic subspaces, to prove the following version of the spectral theorem:

Theorem: If $A$ is a densely-defined self-adjoint operator on Hilbert space $H$ with spectrum $\sigma(A)$, there is a direct-integral Hilbert space $H_{DI}:=\int_{\sigma(A)}^{\oplus} H_{\lambda}\,d\mu$ and a unitary operator $U:H\to H_{DI}$ such that for $L^2$ sections $s\in H_{DI}$ we have $$UAU^{-1}s(\lambda)=\lambda s(\lambda).$$ Then the "nonnormalizable states" here are (after applying $U$) just the elements of the Hilbert spaces $H_{\lambda}$ that are integrated over. Of course it is not possible to actually be in any one of these states, as they have measure zero. But you can take integral superpositions of them as in your expression (1); indeed that is exactly what the space $H_{DI}$ consists of. It is easy to see how this generalizes bound states; in that case, the measure on $H_{DI}$ is a discrete measure and so it is meaningful to discuss $H_{\lambda}$ as an actual element of $H_{DI}$.

This construction is more informative than the standard explanation using multiplication operators and projection-valued measures, as it gives precise meaning to the "nonnormalizable eigenstates" and in particular what it means for them to be degenerate; a degeneracy is just a larger-than-one dimensional $H_{\lambda}$. At the same time, once you're used to it a direct integral space feels (to me at least) more natural and fundamental than a rigged Hilbert space.

Examples:

1. $A=X$, the position operator:

In this case, we simply define $H_{\lambda}$ to be $\mathbb{C}$, and $H_{DI}=\int_{\mathbb{R}}^{\oplus} H_{\lambda}\,d\lambda$, meaning that for any sections $s_1,s_2$ we have $$\langle s_1,s_2\rangle_{H_{DI}}=\int s_1(\lambda)\overline{s_2(\lambda)}\,d\lambda.$$ The map $U$ takes $f\in L^2(\mathbb{R})$ to some section $U(f)$ given by $U(f)(\lambda)=f(\lambda)$. The "eigenstates" $|\lambda\rangle$ of $A$ correspond to the vector $1$ in $H_{\lambda}$ and an integral superposition of $|\lambda\rangle$ is just a section in $H_{DI}$, which can be transmitted back to $H$ by the unitary map $U$.

It may make you squirm a bit to define the $H_{\lambda}$'s based on the pointwise evaluation of an $L^2$ function, which is of course only determined up to measure zero. But the key idea is that so are the $H_{\lambda}$'s; in a direct integral one may throw out measure zero of the underlying Hilbert spaces.

2. $A=-\partial_{x}^2$, the free Hamiltonian:

Here we have $\sigma(A)=[0,\infty)$, and let $H_{\lambda}=\mathbb{C}^2$ for $\lambda>0$. (What $H_0$ is doesn't actually matter as it's measure zero, but we can imagine it to be $\mathbb{C}$.) Define $H_{DI}=\int_0^{\infty}H_{\lambda}\,\frac{1}{2\sqrt\lambda}d\lambda$, meaning that $$\langle s_1,s_2\rangle_{H_{DI}}=\int s_1(\lambda)\cdot\overline{s_2(\lambda)}\,\frac{1}{2\sqrt\lambda}d\lambda.$$ Then any $f\in L^2(\mathbb{R})$ is mapped to a section $U(f)$ with $U(f)(\lambda)=\hat{u}(\sqrt\lambda)\begin{pmatrix}1\\0\end{pmatrix}+\hat{u}(-\sqrt\lambda)\begin{pmatrix}0\\1\end{pmatrix}$ for $\lambda>0$. The "eigenstates" $|\lambda\rangle$ are vectors in the $2$-dimensional eigenstates $H_{\lambda}$ which correspond to "non-normalizable states" $e^{\pm i\sqrt{\lambda}}$. Again, we can recover your (1) from transforming between $H$ and $H_{DI}$ using $U$.

A good exercise at this point to see if you understand this formulation of the spectral theorem is to try it for the Floquet Hamiltonian $H=-\partial_x^2+V$ with $V$ periodic. (See Reed/Simon Chapter 13.16 for an introduction.) It's a bit outside the scope of this answer but is a common type of continuous-spectrum Hamiltonian where we have an intuition of what the "eigenvectors" should be. See if you can make the machinery fit your intuition.

Again, it can make many people very "uncomfortable" to think of Hilbert spaces corresponding to sets of measure zero– we are not used to throwing away entire Hilbert spaces! But after a while it becomes no different than throwing away a set in the definition of an $L^2$ function.