Generalized basis In quantum mechanics, people introduce the notion of "continuous basis" (I actually don't know the mathematical denomination of it). It is not a Schauder basis. I would like to know what could be a good definition of it, and what are the possible difficulties of defining it.
We have a Hilbert space $\mathcal{H}$, and a family of "vectors", $ \left\lbrace|x \rangle \right\rbrace_{x\in \mathbb{R}} $ such that any vector $|\psi \rangle $ in $\mathcal{H}$ can be written
" $|\psi \rangle = \int_{\mathbb{R}} \psi(x)|x \rangle $ "
and
$\int_{\mathbb{R}} |x \rangle \langle x | dx = Id$


*

*We don't say where those $|x \rangle $ live, and in general they are not in $\mathcal{H}$. Usual example from physics $\mathcal{H}=L^2(\mathbb{R})$, and where we think of $|x \rangle$ as the delta distribution.

*It looks a little bit like the spectral theorem that allows to write self adjoint operators as integral with a projector valued measure on the spectrum. I also came across that word, "direct integral" that may have a link.

*we usually also take "orthonormal basis", i.e 
$\langle x |y \rangle = \delta(x-y)$ (what is the meaning of seing it as a distribution in 2 variables)
Of course the decomposition of a vector has to be unique. In the example $\mathcal{H}=L^2(\mathbb{R})$ the coefficients is itself a function $\psi\in L^2(\mathbb{R})$, then unique in the sense "equal almost everywhere" 
Among all the questions such presentation raises:


*

*is the hilbert space structure playing any role? example in defining $\langle x |$ as the "dual basis". Because since those vectors are not even in the hilbert space, their scalar product is not defined.

*is there any difficulty to define an abstract vector space generated by the family $ \left\lbrace|x \rangle \right\rbrace_{x\in \mathbb{R}} $. (Then of course there is still the problem of why a vector from our starting hibert space is equal to some construction in another abstract space.)

*If this whole business is actually well defined, we can see fourier transform as a particular case of writing a vector in the following generalized basis (also from physics)


$|p\rangle:= \left\lbrace x\rightarrow \frac{1}{\sqrt{2\pi}} e^{ipx}\right\rbrace$
and the theorems of Fourier transform being an isometry and being invertible would just say that $|p\rangle $ is an "orthonormale" basis.
 A: You may want to have a look at Gelfand triples. In the example of the impulse operator and the Fourier transform you consider the triple consisting of the Schwartz space, the Hilbert space $L^2(\mathbb{R}^n)$ and the dual of the Schwartz space. The momentum operator is defined on the Schwartz space. The Fourier transform is well-defined on this triple: All the three spaces get mapped to themselves. The waves $e^{ipx}$ are elements of the dual of the Schwartz space and are a complete set of generalised eigen values of the momentum operator. 
Let me sketch the general situation: You have a Hilbert space $H$ and a nuclear topological vector space $S$ linearly embedded into the Hilbert space such that the scalar product is continuous with respect to the topology of $S$. We consider self-adjoint operators $T$ which are defined on $S$. Then generalised eigenvectors as elements of the dual space of $S$ can be defined in the obvious way. Let $\sigma$ be the spectrum of $T$. We can decompose the Hilbert space as a direct integral using a measure $\mu$
$H=\int^\oplus H(\lambda)\mathrm{d}\mu(\lambda)$
such that $T$ acts as a multiplication by $\lambda$ on each space $H(\lambda)$. There exists a unitary operator $U$ mapping $H$ to some space $L^2(X)$ where $T$ acts as a multiplication by a $\sigma$-valued function $a$. It can be proven that for each $x$ in $X$ there exists a $\phi_x$ in the dual space of $S$ such that for every $f\in S$ the functions $Uf\colon X\to\mathbb{C}$ and $x\mapsto\phi_x(f)$ are equal almost everywhere. The functional $\phi_x$ is a generalised eigenvector corresponding to the eigenvalue $a(x)$, that means, given a test function $f\in S$:
$\phi_x(Tf)=a(x)\phi_x(f)$
The space $H(\lambda)$ corresponds, informally spoken, to a space of functions defined on $a^{-1}(\left\{\lambda\right\})$. Thus, by taking linear combinatons, for every element $x\in H(\lambda)$ there is a generalised eigenvector $\phi$ for the eigenvalue $\lambda$. If you choose an orthonormal basis of each space $H(\lambda)$ the corresponding generalised eigenvectors are “complete” in a sense related to the direct integral decomposition.
In the book Introduction to Axiomatic Quantum Field Theory by Bogolubov, Logunov and Todorov you can read a short description, but they do not prove it. They refer to Generalized Functions, Vol. 4: Applications of Harmonic Analysis by Gelfand and Vilenkin.
A: I believe the situation you asked about is addressed by the notion of "rigged Hilbert space"; see for example http://en.wikipedia.org/wiki/Rigged_Hilbert_space .  (If I remember correctly, I came across this notion in the book of Bogolyubov and Shirkov on quantum field theory.)
