I posted this over on MSE without much luck. Not sure if posting here is considered cross-posting but I can remove it if it is.

Let $X\sim\mathcal N(\sqrt 2,1/x^2)$. The expected value $\mathsf EX^{-1}$ is undefined; however, we can assign it a value via the Cauchy principal value interpretation $$ \mathsf{E}X^{-1}\overset{\text{p.v.}}{=}\lim_{\epsilon\nearrow 0}\left(\int_{-\infty}^{-\epsilon}+\int_\epsilon^\infty\right)\frac{1}{t}f_X(t)\,\mathrm dt=\sqrt 2 x\mathcal D(x), $$ where $\mathcal D(x):=e^{-x^2}\int_0^xe^{t^2}\,\mathrm dt$ is Dawson's integral. Notice that as $x\to\infty$, $\mathsf{Var}X\to 0$ and so I wondered what would happen if I were to apply the $\delta$-method to estimating $\mathsf EX^{-1}$. Let $g(X)=1/X$. Expanding $g$ in a Taylor series about $\mathsf EX=\sqrt 2$ gives $$ g(X)=\frac{1}{\sqrt 2}\sum_{k=0}^\infty\left(\frac{\sqrt 2-X}{\sqrt 2}\right)^k $$ and so evaluating the expected value simply requires knowing central moments of the normal distribution. We find $$ \mathsf Eg(X) %=\frac{1}{\sqrt 2}\sum_{k=0}^\infty\mathsf E\left(\frac{\sqrt 2-X}{\sqrt 2}\right)^k %=\frac{1}{\sqrt 2}\sum_{k=0}^\infty\mathsf E\left(\frac{X-\sqrt 2}{\sqrt 2}\right)^{2k} =\frac{1}{\sqrt 2}\sum_{k=0}^\infty\frac{(2k-1)!!}{(2x^2)^k}. $$ Looking back at our expression for $\mathsf EX^{-1}$ it stands to reason that if we then divide $\mathsf Eg(X)$ by $\sqrt 2 x$ that we obtain an estimate for $\mathcal D(x)$ at large $x$, namely, $$ \mathcal D(x)\sim\frac{1}{2x}\sum_{k=0}^\infty\frac{(2k-1)!!}{(2x^2)^k}=\frac{1}{2x}+\frac{1}{4x^3}+\mathcal O(x^{-5}). $$ Comparing these first two terms with eqn. (9) here seems to indicate the above expression is indeed an asymptotic expansion of $\mathcal D(x)$ for large $x$ and thus $\mathsf E g(X)$ provides us with an asymptotic expansion for the Cauchy principal value of $\mathsf EX^{-1}$ as $x\to\infty$.

There is no reason to stop here as we could go further and apply the same approach to "estimate" $\mathsf{Var}X^{-1}$ yielding $$ \mathsf{Var}X^{-1}\sim %\mathsf Eg^2(X)-(\mathsf Eg(X))^2 \frac{1}{2}\sum_{k=0}^\infty\frac{(2k+1)(2k-1)!!}{(2x^2)^k}-\frac{1}{2}\left(\sum_{k=0}^\infty\frac{(2k-1)!!}{(2x^2)^k}\right)^2 =\frac{1}{4x^2}+\frac{1}{x^4}+\mathcal O(x^{-6}). $$

My question fundamentally has to do with

what exactly the $\delta$-method is estimatingwhen we apply it to estimating moments of random variables for which the moments to not exist? In this specific case, we could claim (at least I think) that the $\delta$-method gave us an asymptotic expansion for the Cauchy principal value of $\mathsf EX^{-1}$. However, for the second moment, the Cauchy principal value interpretation of the integral would give us $\mathsf EX^{-2}\overset{\text{p.v.}}{=}\infty$ whereas the $\delta$-method gave us a finite expression. So what the heck did the $\delta$-method estimate?