In fact, there is a very nice object which is suitable for precise mathematical understanding of quantum mechanics including the “delta-function-like” eigenvectors: that of a rigged Hilbert space. It is a Hilbert space $H$ together with a fixed dense inclusion of a locally convex (often assumed nuclear) vector space $\Phi\hookrightarrow H$. An example to think of is the inclusion of the Schwartz space $\mathcal S(\mathbb R^n)$ into $L^2(\mathbb R^n)$.
And indeed, there is a very satisfactory spectral theory of selfadjoint operators on rigged Hilbert spaces which gives, for instance, the precise meaning to the statement “the delta functions $\delta_x$, $x\in[0,1]$ is a complete system of generalised eigenvectors for the operator of multiplication by $x$ on $L^2([0,1],\mathrm{Leb})$”.
A systematic treatment can be found in the classical source
I. M. Gelfand and N. J. Vilenkin. Generalized Functions, vol. 4: Some Applications of Harmonic Analysis. Rigged Hilbert Spaces. Academic Press, New York, 1964.
Applications to some classical problems of quantum mechanics are in the Ph.D. thesis R. de la Madrid, Quantum Mechanics in Rigged Hilbert Space Language (2001).