"Fractional sampling" from a probability distribution

Question

My question concerns an operation on probability distributions which has arisen in some applied research. It is well-defined mathematically (at least in a limited context), but I don't know how to interpret it or express it formally despite the fact that I am well-versed in the language of measure-theoretic probability. I am hoping an interpretation, whether within standard probability theory or not, could lead to ways to extend this idea to other settings or see why it should not extend (maybe it's not internally consistent or something). For simplicity let's assume all probability distributions are discrete or have densities, along with whatever other technical assumptions make you happy.

Suppose we observe independent samples $y_1,\ldots,y_n$ from a parametrized family of distributions $p(y\mid \theta)$ over a finite set $Y$. As per the usual Bayesian framework we assume we have some prior distribution $p(\theta)$ on the parameter and we wish to compute the posterior given the data, $p(\theta \mid y_1,\ldots, y_n)$. The proper thing to do is to normalize with respect to $\theta$ the joint distribution $p(\theta)\prod_{i=1}^n p(y_i\mid \theta)$ evaluated at the data.

Since the observations are in a finite set we could just as easily write this expression as $p(\theta)\prod_{y\in Y} p(y\mid \theta)^{n_y}$, where $n_y$ denotes the number of observations equal to $y$. Now this expression can be evaluated for any $n_y\in [0,\infty)$, not just integers. Since $(p(y\mid \theta)^{1/2})^2 = p(y\mid\theta)$, two "half samples equal to $y$" make a "whole sample equal to $y$". But what is a fractional sample from a probability distribution?

Motivation: In my setting instead of individual samples I have a distribution $p(y)$ over observation values. One reasonable-seeming way to use this to infer $\theta$ is to assign some weight $\beta\in (0,\infty)$ to the observations en masse and let $n_y = \beta p(y)$, which is not necessarily an integer. Even if it is, we do not actually observe at any point $\beta$ "samples" of which $\beta p(y)$ of them were equal to $y$. We only receive the values $p(y)$ for all $y\in Y$. One way of looking at $\beta$ is as a degree of trust in our data $p(y)$ relative to the prior. We could let $\beta$ go to infinity, but this would imply infinite trust in the data relative to the prior $p(\theta)$, which is not appropriate in our case. How to select $\beta$ is another question which I'll ignore for now.

Mathematically, this sort of trick lets us break "samples" into arbitrary small pieces and put them back together again in a consistent way. We could even make these samples infinitesimal and replace the joint distribution evaluated at the data by $p(\theta)\exp\left(\beta\int_Y \log p(y\mid \theta)dp(y)\right)$.

Is there a theory of such fractional or infinitesimal observations? In addition to describing the above, I would expect such a theory to answer questions like, "What is the empirical distribution of $4$ units of mass worth of infinitesimal fair coin flips?" in a consistent manner -- presumably with a continuous probability distribution on $[0,1]$ representing the "observed" fraction of heads. This would be somehow analogous to the role a binomial distribution plays for the usual notion of samples. I have some intuition that perhaps a beta distribution is the right thing, but I can't say why because I'm not sure the question is even meaningful.

I've done a bunch of googling combining words like "fractional" or "partial" or "infinitesimal" with words like "sample" and "observation" and "likelihood" to no avail, probably because there are plenty of more reasonable ways to combine these words. Has such a thing been studied? If so, where can I read about it?

Answer 1

There are surely many directions that you could take this thought, but here is one of them. Note that the formula you have given for the joint distribution of a sample of size $\beta$ with a continuous empirical distribution closely resembles the definition the relative entropy of a distribution $\rho$ with respect to another distribution $\mu$: $$H(\rho | \mu) = \int\log(d\rho/d\mu)d\rho. $$ In fact, for any sample $\{x_i\}$ of size $n$ if we let $p$ denote the empirical distribution $\frac{1}{n}\sum \delta(x_i)$, then the log joint density is $$\sum \log(d\mu(x_i))=n\int \log(d\mu)dp.$$ This formula gives you joint probability density at a given point in $\mathbb R^n$, but it doesn't accurately measure the "probability" of the resulting empirical distribution, because it fails to take into account the possible permutations of the sample points. Using Stirling's approximation, and letting $n_i$ denote the number of sample points at each distinct value, we may asymptotically approximate the number of permutations which yield the same empirical distribution $$\log\frac{n!}{n_1!n_2!\cdots n_k!}\simeq n\log n - \sum n_i\log n_i = -n \left(\sum p_i \log p_i\right). $$

Putting these two formulas together yields an estimate of the "probability" of generating a specific empirical distribution (with considerable abuse of notation): $$n\int \left(\log(d\mu)-\log(p)\right)dp. $$ Note that we can't write this in terms of relative entropy as $-nH(p| \mu)$, because $p$ is not absolutely continuous with respect to $\mu$, so the Radon-Nikodym derivative is not well defined. However, if the empirical distribution $p$ were in fact a continuous distribution, as you have in your research problem, then this formula would begin to make some sense.

These ideas have in fact been made rigorous in the theory of large deviations. Let $\mathcal P (\mathbb R)$ denote the set of all probability distributions on $\mathbb R$, under the topology of weak convergence. Let $\mu\in \mathcal P(\mathbb R)$ be any probability distribution, and let $\mu_n$ be a random element in $\mathcal P(\mathbb R)$ corresponding to the empirical distribution on an i.i.d sample of size $n$ drawn according to $\mu$.

As $n\rightarrow\infty$, $\mu_n$ will converge in probability to $\mu$, and the rate of this convergence can be described by a large deviations principle, for which the rate function is the relative entropy with respect to $\mu$. What this means is that, for any measurable set $B\in \mathcal P(\mathbb R)$,

$$\begin{align} \liminf_{n\rightarrow \infty}\frac{1}{n} \log P[\mu_n\in B]&\geq -\inf_{\rho\in B^o}H(\rho|\mu)\\ \liminf_{n\rightarrow \infty}\frac{1}{n} \log P[\mu_n\in B]&\leq -\inf_{\rho\in \bar{B}}H(\rho|\mu), \end{align}$$ where $B^o$ and $\bar{B}$ are the interior and closure of $B$.

So this formalizes the idea that the probability of drawing a sample of size $n$ from $\mu$ and ending up with an empirical distribution in the neighborhood of $\rho$ is approximately $e^{-nH(\rho|\mu)}$.

As far as what a "fractional sample" actually means, I'm not sure it matters any more than what the "factorial" of a non-integer is, or what the "square root" of a negative number is, provided that it is a useful mathematical construct.

Answer 2

I am not sure about the interpretation of half samples, but in the motivation part, what you are doing when writing $p(\theta)\exp(\beta\int\log p(y|\theta) dp(y))$ is construct a measure on $\theta$ with $\beta$ playing the role of inverse temperature. $\beta=\infty$ (zero temperature) would correspond to the maximum likelihood. When you take i.i.d. observations, what you have is $p(\theta)\exp(\beta n <L_n, \log p(\cdot|\theta)>)$, where $L_n$ is the empirical measure of the observations. This is the link with large deviations that @mpr points out. $\beta$ will then show up as a multiplier in optimization problems related to integration over $\theta$ of cost functions of the form $e^{n g(\theta)}$.