Measure of bounded fourth (and below) moment distributions?

Question

Many results in probability theory/random matrix theory/etc require probability distributions with finite fourth moments; what is the measure of such probability distributions (in the space of probability measures)? While the question "what RVs has finite fourth moment" is answered here, I'm trying to determine how often you would expect "real-life data" to conform to hypotheses in the various results above.

Edit: since this isn't my primary field, my question is naive, apologies. I was primarily interested in the measure of finite moment distributions relative to infinite ones, particularly fourth and lower moments (although I suspect it won't matter). However, thanks to Nate, below, I now know there is no canonical measure on distributions. Since there are no canonical measures, my question becomes:

What are some commonly-used measures for the space of distributions, and what proportion do finite moment distributions occupy in them?

Additionally, because there aren't canonical measures, I'd like to know if these finite moment distributions are dense in two particular topologies, the ones induced by $L_p$ and Fisher metrics (generally divergence-induced metrics). (I haven't though about how the two metrics are related topologically, if at all).

Answer 1

I'll answer the topological version. Let's work in one dimension for simplicity, so we consider the space $\newcommand{\PR}{\mathcal{P}(\mathbb{R})}\PR$ of probability measures on $\mathbb{R}^1$.

Most of the commonly encountered metrics on $\PR$ induce the same topology: the weak topology. The standard reference for facts about this topology is Billingsley, Convergence of Probability Measures. The basic definition is that a sequence of measures $\mu_n$ converges weakly to a measure $\mu$ if $\int f\,d\mu_n \to \int f\,d\mu$ for all bounded continuous functions $f : \mathbb{R} \to \mathbb{R}$. However, for our purposes, a convenient fact is that when working on $\mathbb{R}$ (or any separable locally compact metric space), "bounded continuous" can be replaced with "continuous and compactly supported".

I claim the set of measures having bounded support is dense in $\PR$. (In particular all such measures have finite fourth moment.) To see this, fix a measure $\mu \in \PR$ and let $\mu_n(A) = \frac{\mu(A \cap [-n,n])}{\mu([-n,n])}$. Clearly $\mu_n$ is a probability measure with bounded support. Let $f \in C_c(\mathbb{R})$ and choose $N$ so large that $f$ is supported in $[-N,N]$. Then for all $n \ge N$ we have $$\int f\,d\mu_n = \frac{1}{\mu([-n,n])} \int f\,d\mu \to \int f\,d\mu$$ as $n \to \infty$. So $\mu_n \to \mu$ weakly.

On the other hand, the space of probability measures $\newcommand{\PF}{\mathcal{P}^4(\mathbb{R})}\PF$ with finite fourth moment is meager in the weak topology. Let $A_k$ be the set of all probability measures having (uncentered) fourth moment $\le k$, so that $\PF = \bigcup_n A_k$. I claim $A_k$ is closed and nowhere dense.

Let $\mu_n \in A_k$ and suppose $\mu_n \to \mu$ weakly. Set $f_j(x) = \min(x^4, j)$ so that $f_j$ is bounded and continuous. We have $\int f_j\,d\mu_n \le \int x^4\,d\mu_n \le k$ for all $j,n$. Letting $n \to \infty$ we have $\int f_j\,d\mu = \lim_{n \to \infty} \int f_j\,d\mu_n \le k$. Now letting $j\to \infty$, we have $\int x^4\,d\mu = \lim_{j \to \infty} \int f_j\,d\mu$ by monotone convergence, so $\int x^4\,d\mu \le k$ and we have $\mu \in A_k$. Thus $A_k$ is closed.

Suppose $\mu \in A_k$ and let $\mu_n = (1 - \frac{1}{n}) \mu + \frac{1}{n} \delta_n$ where $\delta_n$ is a point mass at $n$. An argument like the above shows that $\mu_n \to \mu$ weakly, but you can also see that $\int x^4\,d\mu_n \ge n^3$, so for sufficiently large $n$ we have $\mu_n \notin A_k$. Thus $\mu$ is not in the interior of $A_k$; $\mu$ was arbitrary so $A_k$ is nowhere dense.