67
$\begingroup$

If two different probability distributions have identical moments, are they equal? I suspect not, but I would guess they are "mostly" equal, for example, on everything but a set of measure zero. Does anyone know an example of two different probability distributions with identical moments? The less pathological the better. Edit: Is it unconditionally true if I specialize to discrete distributions?

And a related question: Suppose I ask the same question about Renyi entropies. Recall that the Renyi entropy is defined for all $a \geq 0$ by

$$ H_a(p) = \frac{\log \left( \sum_j p_j^a \right)}{1-a} $$

You can define $a = 0, 1, \infty$ by taking suitable limits of this formula. Are two distributions with identical Renyi entropies (for all values of the parameter $a$) actually equal? How "rigid" is this result? If I allow two Renyi entropies of distributions $p$ and $q$ to differ by at most some small $\epsilon$ independent of $a$, then can I put an upper bound on, say, $||p-q||_1$ in terms of $\epsilon$? What can be said in the case of discrete distributions?

$\endgroup$

9 Answers 9

51
$\begingroup$

Roughly speaking, if the sequence of moments doesn't grow too quickly, then the distribution is determined by its moments. One sufficient condition is that if the moment generating function of a random variable has positive radius of convergence, then that random variable is determined by its moments. See Billingsley, Probability and Measure, chapter 30.

A standard example of two distinct distributions with the same moment is based on the lognormal distribution:

$$f_0(x) = \sqrt{2 \pi} x^{-1} e^{-(\log x)^2 /2}$$

which is the density of the lognormal, and the perturbed version

$$f_a(x) = f_0(x)(1 + a \sin(2 \pi \log x))$$

These have the same moments; namely the nth moment of each of these is exp(n2/2).

A condition for a distribution over the reals to be determined by its moments is that lim supk → ∞2k)1/2k/2k is finite, where μ2k is the (2k)th moment of the distribution. For a distribution supported on the positive reals, lim supk → ∞k)1/2k/2k being finite suffices.

This example is from Rick Durrett, Probability: Theory and Examples, 3rd edition, pp. 106-107; as the original source for the lognormal Durrett cites C. C. Heyde (1963) On a property of the lognormal distribution, J. Royal. Stat. Soc. B. 29, 392-393.

$\endgroup$
3
  • 1
    $\begingroup$ Interesting that you get a weaker condition for distributions supported on the positive reals, but I can see why that should be so. If X is a non-negative random variable then you can set Y=ε√X, where ε=+1,-1 each with probability 1/2 and independent of X. Then, the odd moments of Y are zero, the 2n'th moment of Y equals the n'th moment of X, and determining the distribution of Y is enough to find the distribution of X. $\endgroup$ Oct 31, 2009 at 17:15
  • $\begingroup$ Thanks Michael! Do you know anything about the "Renyi" part of the question? $\endgroup$ Oct 31, 2009 at 20:08
  • $\begingroup$ Steve, I don't know anything about the Renyi part. I wish I did. $\endgroup$ Nov 3, 2009 at 1:42
20
$\begingroup$

As has been mentioned in previous answers, the moments do not uniquely determine the distributions unless certain conditions are satisfied, such as bounded distributions. One thing you can say, is that the distribution of a random variable $X$ is uniquely determined by the characteristic function $\phi_X(a)=E[\exp(iaX)]$. Letting $m_n=E[X^n]$ be the $n^{th}$ moment, this can be expanded as

$$\phi_X(a)=\Sigma_n \frac{i^na^nX^n}{n!}$$

which is valid within its radius of convergence. So, the moments will uniquely determine the distribution as long as this has infinite radius of convergence, which is the case as long as $$\lim_{n\rightarrow\infty}\sup\left|\frac{m_n}{n!}\right|^{\frac{1}{n}}=0.$$ Stirling's formula simplifies it a bit to limsupn→∞|mn|1/n/n=0. This can be proven using the dominated convergence theorem.

For example, a distribution is bounded by K if |mn|≤Kn, which satisfies this condition.

On the other hand, it is possible to construct distinct distributions supported in the positive integers and with the same moments. To do this, you need to find a sequence of real numbers cn satisfying Σncnnr=0 for all r (and converging absolutely). This doesn't involve anything more than solving some linear equations to solve this for any finite set of powers r. Then, by keeping adding more terms to extend to all positive integers r, you get the infinite sequence cn. The two distributions can then be obtained by taking the positive and negative parts of cn.

$\endgroup$
9
$\begingroup$

This sounds like one of the classical "moment problems" that have been much studied, although I'm afraid I don't know the literature. Wikipedia suggests that the term to look for is Hamburger moment problem

A quick Google also throws up an article by Stoyanov which ought to have some examples of non-uniqueness and pointers to the literature.

As you might know, if we know in advance that the density is confined to some bounded interval (say [-1,1] for sake of argument), then the moments do indeed determine the density. (This basically follows because the density is determined by its values when integrated against continuous functions, and continuous functions on a closed bounded interval can be approximated to arbitrary accuracy by polynomials)

$\endgroup$
7
$\begingroup$

Suppose all moments exist for X and Y.

1) If X and Y have bounded support, the CDFs of X and Y are equal if and only if all moments are equal.

2) If the moment generating functions exist and M_X(t) = M_Y(t) for all t in an open neighborhood of 0, then the CDFs of X and Y are equal.

$\endgroup$
2
  • $\begingroup$ Can you give a reference or an explanation for why the second result is true? I believe Yemon explained why the first one is true. $\endgroup$ Oct 31, 2009 at 14:58
  • 1
    $\begingroup$ Here's a reference: Statistical Inference by Casella and Berger (bit.ly/2ZMNV0) 2nd edition, page 65. $\endgroup$ Oct 31, 2009 at 16:11
6
$\begingroup$

Regarding your second question:

The Renyi entropy depends only on the probabilities, and not on the values the RV take; any 1-1 function of the RV have the same entropy.

If you's asking whether the Renyi entropy determines the sequence of probabilities pi, then the answer is yes. Assume WLOG that pi are in descending order. Then the limit when a tends to infinity, of Ha is p0. Once you know p0, it is easy to calculate the entropy for the sequence p1, p2,.. which then allows us to find p1, etc.

$\endgroup$
1
  • $\begingroup$ For what it's worth, this proof appears as Proposition A22 in the appendix to Tom Leinster and Christina Cobbold, Measuring diversity: the importance of species similarity, Ecology 93 (2012), 477-489. And it's also Lemma A.3.1 in my book Entropy and Diversity: The Axiomatic Approach (maths.ed.ac.uk/~tl/ed). But it's so simple that surely someone noticed before either of us --- Rényi himself, probably. $\endgroup$ Oct 5, 2021 at 23:55
5
$\begingroup$

I don't have it on hand, but Billingsley's book "Probability and Measure" has a nice section on this issue, including the classic example of a distribution not uniquely determined by its moments: the log-normal distribution (i.e., the distribution of e^Z, where Z~N(0,1)).

There are known (but not to me off the top of my head) necessary and sufficient conditions for a distribution to be determined by its moments, in terms of the rate of growth of the moments; I think but I'm not sure those are in Billingsley. If not, I'd check Feller next. In any case, I expect that the situation is not better for discrete distributions - you can discretize the log-normal distribution in a way that increases the size of the moments to get a discrete distribution. Then you get a discrete probability distribution with the same moments as some other probability distribution. I don't know a proof that you can arrange for the second distribution also to be discrete, but I'd guess you can.

As for your second question, unless I'm misunderstanding something then I think a discrete counterexample to the first question also provides a counterexample to the second.

$\endgroup$
1
  • 1
    $\begingroup$ Billingsley says that a sufficient condition for a distribution to be determined by its moments is that the moment generating function converges. That means that the moments grow at most exponentially fast. $\endgroup$ Oct 31, 2009 at 15:39
5
$\begingroup$

Thinking about the Renyi part of this question again today, I realized that there is a simple and elegant way to show the equivalence of knowing the Renyi entropies and knowing the probabilities (in principle) without taking limits. See Ori's comments, also.

Suppose we have just a finite number outcomes. Then we can place all of the probabilities for each outcome on the diagonal of a large matrix. The Renyi entropies are basically just the traces of the powers of this matrix for integer values of $\alpha$. We would like to show that knowing these trace powers is equivalent to knowing the probabilities themselves. Intuitively, this seems clear, since it is just an overdetermined system of polynomial equations, but a priori it isn't clear that there isn't some weird degeneracy hidden somewhere that would preclude a unique solution. So, we have the trace powers, and as a function of the probabilities, these are just the power sums. We can use the Newton-Girard identities to transform these into the elementary symmetric polynomials. Then we can express the characteristic polynomial of our large matrix as a sum over these. The roots of this polynomial are of course the eigenvalues, which are just the probabilities in question.

$\endgroup$
2
  • $\begingroup$ Very nice. I'm not sure how does this method work for the infinite support case. Notice also that when taking limits you only need to know the (exact) tail behavior of the Renyi entropy. $\endgroup$ Nov 10, 2009 at 1:53
  • $\begingroup$ Yes, it is not clear how to make this work in the infinite support case. For the finite case, I just think it is nice because you only use as many values of the Renyi entropy as there are outcomes in the probability vector. So it feels somehow very economical and more satisfying than taking limits. However I agree with you that, for any actual computation, this is much harder to work with than taking limits. $\endgroup$ Nov 10, 2009 at 2:34
3
$\begingroup$

I stumbled across this post while googling a question about moment determinacy. Then I also found this survey (Recent Developments on the Moment Problem, Gwo Dong Lin), which summarizes pretty much everything said here on the moment problem and gathers a lot more checkable conditions for moment (in)determinacy of probability distributions. I found it to be a great resource.

$\endgroup$
0
$\begingroup$

I've heard (from my undergraduate stats profs) the answer is that 2 distributions can have the same moments but different distributions. I either don't remember or never had an actual example though. I'd guess you could (maybe) look for an example by camparing a discrete distribution and a continous one.

$\endgroup$
1
  • 4
    $\begingroup$ There are distributions that differ from each other while sharing the same moments. But no distribution but a normal distribution has the same moments as a normal distribution, and there are other distributions of which the same is true. $\endgroup$ Jun 18, 2020 at 4:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.