When are probability distributions completely determined by their moments?

Question

If two different probability distributions have identical moments, are they equal? I suspect not, but I would guess they are "mostly" equal, for example, on everything but a set of measure zero. Does anyone know an example of two different probability distributions with identical moments? The less pathological the better. Edit: Is it unconditionally true if I specialize to discrete distributions?

And a related question: Suppose I ask the same question about Renyi entropies. Recall that the Renyi entropy is defined for all $a \geq 0$ by

$$ H_a(p) = \frac{\log \left( \sum_j p_j^a \right)}{1-a} $$

You can define $a = 0, 1, \infty$ by taking suitable limits of this formula. Are two distributions with identical Renyi entropies (for all values of the parameter $a$) actually equal? How "rigid" is this result? If I allow two Renyi entropies of distributions $p$ and $q$ to differ by at most some small $\epsilon$ independent of $a$, then can I put an upper bound on, say, $||p-q||_1$ in terms of $\epsilon$? What can be said in the case of discrete distributions?

Answer 1

Roughly speaking, if the sequence of moments doesn't grow too quickly, then the distribution is determined by its moments. One sufficient condition is that if the moment generating function of a random variable has positive radius of convergence, then that random variable is determined by its moments. See Billingsley, Probability and Measure, chapter 30.

A standard example of two distinct distributions with the same moment is based on the lognormal distribution:

$$f_0(x) = \sqrt{2 \pi} x^{-1} e^{-(\log x)^2 /2}$$

which is the density of the lognormal, and the perturbed version

$$f_a(x) = f_0(x)(1 + a \sin(2 \pi \log x))$$

These have the same moments; namely the nth moment of each of these is exp(n²/2).

A condition for a distribution over the reals to be determined by its moments is that lim sup_{k → ∞} (μ_2k)^1/2k/2k is finite, where μ_2k is the (2k)th moment of the distribution. For a distribution supported on the positive reals, lim sup_{k → ∞} (μ_k)^1/2k/2k being finite suffices.

This example is from Rick Durrett, Probability: Theory and Examples, 3rd edition, pp. 106-107; as the original source for the lognormal Durrett cites C. C. Heyde (1963) On a property of the lognormal distribution, J. Royal. Stat. Soc. B. 29, 392-393.

Answer 2

As has been mentioned in previous answers, the moments do not uniquely determine the distributions unless certain conditions are satisfied, such as bounded distributions. One thing you can say, is that the distribution of a random variable $X$ is uniquely determined by the characteristic function $\phi_X(a)=E[\exp(iaX)]$. Letting $m_n=E[X^n]$ be the $n^{th}$ moment, this can be expanded as

$$\phi_X(a)=\Sigma_n \frac{i^na^nX^n}{n!}$$

which is valid within its radius of convergence. So, the moments will uniquely determine the distribution as long as this has infinite radius of convergence, which is the case as long as $$\lim_{n\rightarrow\infty}\sup\left|\frac{m_n}{n!}\right|^{\frac{1}{n}}=0.$$ Stirling's formula simplifies it a bit to limsup_n→∞|m_n|^1/n/n=0. This can be proven using the dominated convergence theorem.

For example, a distribution is bounded by K if |m_n|≤Kⁿ, which satisfies this condition.

On the other hand, it is possible to construct distinct distributions supported in the positive integers and with the same moments. To do this, you need to find a sequence of real numbers c_n satisfying Σ_nc_nn^r=0 for all r (and converging absolutely). This doesn't involve anything more than solving some linear equations to solve this for any finite set of powers r. Then, by keeping adding more terms to extend to all positive integers r, you get the infinite sequence c_n. The two distributions can then be obtained by taking the positive and negative parts of c_n.

Answer 3

This sounds like one of the classical "moment problems" that have been much studied, although I'm afraid I don't know the literature. Wikipedia suggests that the term to look for is Hamburger moment problem

A quick Google also throws up an article by Stoyanov which ought to have some examples of non-uniqueness and pointers to the literature.

As you might know, if we know in advance that the density is confined to some bounded interval (say [-1,1] for sake of argument), then the moments do indeed determine the density. (This basically follows because the density is determined by its values when integrated against continuous functions, and continuous functions on a closed bounded interval can be approximated to arbitrary accuracy by polynomials)

Answer 4

Suppose all moments exist for X and Y.

1) If X and Y have bounded support, the CDFs of X and Y are equal if and only if all moments are equal.

2) If the moment generating functions exist and M_X(t) = M_Y(t) for all t in an open neighborhood of 0, then the CDFs of X and Y are equal.

Answer 5

Regarding your second question:

The Renyi entropy depends only on the probabilities, and not on the values the RV take; any 1-1 function of the RV have the same entropy.

If you's asking whether the Renyi entropy determines the sequence of probabilities p_i, then the answer is yes. Assume WLOG that p_i are in descending order. Then the limit when a tends to infinity, of H_a is p₀. Once you know p₀, it is easy to calculate the entropy for the sequence p₁, p₂,.. which then allows us to find p₁, etc.

Answer 6

Thinking about the Renyi part of this question again today, I realized that there is a simple and elegant way to show the equivalence of knowing the Renyi entropies and knowing the probabilities (in principle) without taking limits. See Ori's comments, also.

Suppose we have just a finite number outcomes. Then we can place all of the probabilities for each outcome on the diagonal of a large matrix. The Renyi entropies are basically just the traces of the powers of this matrix for integer values of $\alpha$. We would like to show that knowing these trace powers is equivalent to knowing the probabilities themselves. Intuitively, this seems clear, since it is just an overdetermined system of polynomial equations, but a priori it isn't clear that there isn't some weird degeneracy hidden somewhere that would preclude a unique solution. So, we have the trace powers, and as a function of the probabilities, these are just the power sums. We can use the Newton-Girard identities to transform these into the elementary symmetric polynomials. Then we can express the characteristic polynomial of our large matrix as a sum over these. The roots of this polynomial are of course the eigenvalues, which are just the probabilities in question.

Answer 7

I don't have it on hand, but Billingsley's book "Probability and Measure" has a nice section on this issue, including the classic example of a distribution not uniquely determined by its moments: the log-normal distribution (i.e., the distribution of e^Z, where Z~N(0,1)).

There are known (but not to me off the top of my head) necessary and sufficient conditions for a distribution to be determined by its moments, in terms of the rate of growth of the moments; I think but I'm not sure those are in Billingsley. If not, I'd check Feller next. In any case, I expect that the situation is not better for discrete distributions - you can discretize the log-normal distribution in a way that increases the size of the moments to get a discrete distribution. Then you get a discrete probability distribution with the same moments as some other probability distribution. I don't know a proof that you can arrange for the second distribution also to be discrete, but I'd guess you can.

As for your second question, unless I'm misunderstanding something then I think a discrete counterexample to the first question also provides a counterexample to the second.

Answer 8

I stumbled across this post while googling a question about moment determinacy. Then I also found this survey (Recent Developments on the Moment Problem, Gwo Dong Lin), which summarizes pretty much everything said here on the moment problem and gathers a lot more checkable conditions for moment (in)determinacy of probability distributions. I found it to be a great resource.

Answer 9

I've heard (from my undergraduate stats profs) the answer is that 2 distributions can have the same moments but different distributions. I either don't remember or never had an actual example though. I'd guess you could (maybe) look for an example by camparing a discrete distribution and a continous one.