Perhaps an example will help? Consider the momentum operator $A=-id/dx$, the eigenstates $|a\rangle$ are plane waves $\langle x|a\rangle=(2\pi)^{-1/2} e^{iax}$, the expansion (1) of an arbitrary state $|\psi\rangle$ in the $|a\rangle$ basis is the Fourier integral
$$\langle x|\psi\rangle=\int_{-\infty}^\infty da\,\psi(a)\langle x|a\rangle=(2\pi)^{-1/2}\int_{-\infty}^\infty da\,e^{iax}\psi(a),$$
and the completeness relation reads
$$\int_{-\infty}^\infty da\,\langle x|a\rangle\langle a|x'\rangle=(2\pi)^{-1}\int_{-\infty}^\infty da\,e^{ia(x-x')}=\delta(x-x')=\langle x|I|x'\rangle,$$
with $\delta(x-x')$ the Dirac delta function distribution.

So yes, you need to equip the Hilbert space with a distribution theory --- as for any unbounded observable with a continuous spectrum a rigged Hilbert space is needed.

I found this <A HREF="https://arxiv.org/abs/quant-ph/0502053">discussion</A> of the Dirac bra-ket formalism in a continuous spectrum instructive.