There is a very nice setup which is suitable for precise mathematical understanding of quantum mechanics including the “delta-function-like eigenvectors”: that of a rigged Hilbert space. It is a Hilbert space $H$ together with a fixed dense continuous inclusion of a locally convex (often assumed nuclear) topological vector space $\Phi\hookrightarrow H$. An example to think of is the inclusion of the Schwartz space $\mathcal S(\mathbb R^n)$ into $L^2(\mathbb R^n)$.
And indeed, there is a very satisfactory spectral theory of selfadjoint operators on rigged Hilbert spaces which gives, for instance, the precise meaning to the statement “the delta functions $\delta_x$, $x\in[0,1]$ form a complete system of generalised eigenvectors for the operator of multiplication by $x$ on $L^2([0,1],\mathrm{Leb})$”.
A systematic treatment can be found in the classical source
I. M. Gelfand and N. J. Vilenkin. Generalized Functions, vol. 4: Some Applications of Harmonic Analysis. Rigged Hilbert Spaces. Academic Press, New York, 1964.
Applications to some classical problems of quantum mechanics can be found in the Ph.D. thesis R. de la Madrid, Quantum Mechanics in Rigged Hilbert Space Language (2001).