There is a very nice setup which is suitable for precise mathematical understanding of quantum mechanics including the “delta-function-like eigenvectors”: that of a [rigged Hilbert space][1]. It is a Hilbert space $H$ together with a fixed dense continuous inclusion of a locally convex (often assumed nuclear) topological vector space $\Phi\hookrightarrow H$. An example to think of is the inclusion of the Schwartz space $\mathcal S(\mathbb R^n)$ into $L^2(\mathbb R^n)$.

And indeed, there is a very satisfactory spectral theory of selfadjoint operators on rigged Hilbert spaces which gives, for instance, the precise meaning to the statement “the delta functions $\delta_x$, $x\in[0,1]$ form a complete system of generalised eigenvectors for the operator of multiplication by $x$ on $L^2([0,1],\mathrm{Leb})$”.

A systematic treatment can be found in the classical source

I. M. Gelfand and N. J. Vilenkin. _Generalized Functions, vol. 4: Some Applications of Harmonic Analysis. Rigged Hilbert Spaces._ Academic Press, New York, 1964.

Applications to some classical problems of quantum mechanics can be found in the Ph.D. thesis [R. de la Madrid, _Quantum Mechanics in Rigged Hilbert Space Language_ (2001)][2]. 


  [1]: https://en.wikipedia.org/wiki/Rigged_Hilbert_space
  [2]: http://galaxy.cs.lamar.edu/~rafaelm/webdis.pdf