When are probability distributions completely determined by their moments? - MathOverflow most recent 30 from http://mathoverflow.net 2013-05-24T14:47:21Z http://mathoverflow.net/feeds/question/3525 http://www.creativecommons.org/licenses/by-nc/2.5/rdf http://mathoverflow.net/questions/3525/when-are-probability-distributions-completely-determined-by-their-moments When are probability distributions completely determined by their moments? Steve Flammia 2009-10-31T09:57:27Z 2009-11-09T23:49:12Z <p>If two different probability distributions have identical moments, are they equal? I suspect not, but I would guess they are "mostly" equal, for example, on everything but a set of measure zero. Does anyone know an example of two different probability distributions with identical moments? The less pathological the better. <strong>Edit:</strong> Is it unconditionally true if I specialize to discrete distributions?</p> <p>And a related question: Suppose I ask the same question about Renyi entropies. Recall that the Renyi entropy is defined for all <code>a</code> &ge; 0 by</p> <p>H<sub>a</sub>(p) = log(&sum;<sub>j</sub> p<sub>j</sub><sup>a</sup>)/(1-a)</p> <p>You can define <code>a</code>=0,1,&infin; by taking suitable limits of this formula. Are two distributions with identical Renyi entropies (for all values of the parameter <code>a</code>) actually equal? How "rigid" is this result? If I allow two Renyi entropies of distributions <code>p</code> and <code>q</code> to differ by at most some small &epsilon; independent of <code>a</code>, then can I put an upper bound on, say, || p - q ||<sub>1</sub> in terms of &epsilon;? What can be said in the case of discrete distributions?</p> http://mathoverflow.net/questions/3525/when-are-probability-distributions-completely-determined-by-their-moments/3526#3526 Answer by Yemon Choi for When are probability distributions completely determined by their moments? Yemon Choi 2009-10-31T10:06:33Z 2009-10-31T10:06:33Z <p>This sounds like one of the classical "moment problems" that have been much studied, although I'm afraid I don't know the literature. Wikipedia suggests that the term to look for is <a href="http://en.wikipedia.org/wiki/Hamburger%5Fmoment%5Fproblem" rel="nofollow">Hamburger moment problem</a></p> <p>A quick Google also throws up an <a href="http://projecteuclid.org/euclid.bj/1081282696" rel="nofollow">article by Stoyanov</a> which ought to have some examples of non-uniqueness and pointers to the literature.</p> <p>As you might know, if we know in advance that the density is confined to some bounded interval (say [-1,1] for sake of argument), then the moments do indeed determine the density. (This basically follows because the density is determined by its values when integrated against continuous functions, and continuous functions on a closed bounded interval can be approximated to arbitrary accuracy by polynomials)</p> http://mathoverflow.net/questions/3525/when-are-probability-distributions-completely-determined-by-their-moments/3536#3536 Answer by John D. Cook for When are probability distributions completely determined by their moments? John D. Cook 2009-10-31T12:14:38Z 2009-10-31T12:14:38Z <p>Suppose all moments exist for X and Y.</p> <p>1) If X and Y have bounded support, the CDFs of X and Y are equal if and only if all moments are equal.</p> <p>2) If the moment generating functions exist and M_X(t) = M_Y(t) for all t in an open neighborhood of 0, then the CDFs of X and Y are equal.</p> http://mathoverflow.net/questions/3525/when-are-probability-distributions-completely-determined-by-their-moments/3549#3549 Answer by Jonathan Kariv for When are probability distributions completely determined by their moments? Jonathan Kariv 2009-10-31T14:58:56Z 2009-10-31T14:58:56Z <p>I've heard (from my undergraduate stats profs) the answer is that 2 distributions can have the same moments but different distributions. I either don't remember or never had an actual example though. I'd guess you could (maybe) look for an example by camparing a discrete distribution and a continous one.</p> http://mathoverflow.net/questions/3525/when-are-probability-distributions-completely-determined-by-their-moments/3552#3552 Answer by Mark Meckes for When are probability distributions completely determined by their moments? Mark Meckes 2009-10-31T15:22:39Z 2009-10-31T15:22:39Z <p>I don't have it on hand, but Billingsley's book "Probability and Measure" has a nice section on this issue, including the classic example of a distribution not uniquely determined by its moments: the log-normal distribution (i.e., the distribution of e^Z, where Z~N(0,1)).</p> <p>There are known (but not to me off the top of my head) necessary and sufficient conditions for a distribution to be determined by its moments, in terms of the rate of growth of the moments; I think but I'm not sure those are in Billingsley. If not, I'd check Feller next. In any case, I expect that the situation is not better for discrete distributions - you can discretize the log-normal distribution in a way that increases the size of the moments to get a discrete distribution. Then you get a discrete probability distribution with the same moments as some other probability distribution. I don't know a proof that you can arrange for the second distribution also to be discrete, but I'd guess you can.</p> <p>As for your second question, unless I'm misunderstanding something then I think a discrete counterexample to the first question also provides a counterexample to the second.</p> http://mathoverflow.net/questions/3525/when-are-probability-distributions-completely-determined-by-their-moments/3553#3553 Answer by Michael Lugo for When are probability distributions completely determined by their moments? Michael Lugo 2009-10-31T15:38:08Z 2009-10-31T15:38:08Z <p>Roughly speaking, if the sequence of moments doesn't grow too quickly, then the distribution is determined by its moments. One sufficient condition is that if the moment generating function of a random variable has positive radius of convergence, then that random variable is determined by its moments. See Billingsley, <i>Probability and Measure</i>, chapter 30.</p> <p>A standard example of two distinct distributions with the same moment is based on the lognormal distribution:</p> <p>f<sub>0</sub>(x) = (2&pi;)<sup>1/2</sup> x<sup>-1</sup> exp(-(log x)<sup>2</sup>/2).</p> <p>which is the density of the lognormal, and the perturbed version</p> <p>f<sub>a</sub>(x) = f<sub>0</sub>(x) (1 + a sin (2&pi; log x))</p> <p>These have the same moments; namely the nth moment of each of these is exp(n<sup>2</sup>/2). </p> <p>A condition for a distribution over the reals to be determined by its moments is that lim sup<sub>k &rarr; &infin;</sub> (&mu;<sub>2k</sub>)<sup>1/2k</sup>/2k is finite, where &mu;<sub>2k</sub> is the (2k)th moment of the distribution. For a distribution supported on the positive reals, lim sup<sub>k &rarr; &infin;</sub> (&mu;<sub>k</sub>)<sup>1/2k</sup>/2k being finite suffices.</p> <p>This example is from Rick Durrett, <i>Probability: Theory and Examples</i>, 3rd edition, pp. 106-107; as the original source for the lognormal Durrett cites C. C. Heyde (1963) On a property of the lognormal distribution, J. Royal. Stat. Soc. B. 29, 392-393.</p> http://mathoverflow.net/questions/3525/when-are-probability-distributions-completely-determined-by-their-moments/3556#3556 Answer by George Lowther for When are probability distributions completely determined by their moments? George Lowther 2009-10-31T15:49:18Z 2009-10-31T16:07:20Z <p>As has been mentioned in previous answers, the moments do not uniquely determine the distributions unless certain conditions are satisfied, such as bounded distributions. One thing you can say, is that the distribution of a random variable X is uniquely determined by the characteristic function &phi;<sub>X</sub>(a)=E[exp(iaX)]. Letting m<sub>n</sub>=E[X<sup>n</sup>] be the n'th moment, this can be expanded as</p> <p>&phi;<sub>X</sub>(a)&nbsp;=&nbsp;&Sigma;<sub>n</sub>i<sup>n</sup>a<sup>n</sup>m<sub>n</sub>/n!</p> <p>which is valid within its radius of convergence. So, the moments will uniquely determine the distribution as long as this has infinite radius of convergence, which is the case as long as limsup<sub>n&rarr;&infin;</sub>|m<sub>n</sub>/n!|<sup>1/n</sup>=0. Stirling's formula simplifies it a bit to limsup<sub>n&rarr;&infin;</sub>|m<sub>n</sub>|<sup>1/n</sup>/n=0. This can be proven using the dominated convergence theorem.</p> <p>For example, a distribution is bounded by K if |m<sub>n</sub>|&le;K<sup>n</sup>, which satisfies this condition.</p> <p>On the other hand, it is possible to construct distinct distributions supported in the positive integers and with the same moments. To do this, you need to find a sequence of real numbers c<sub>n</sub> satisfying &Sigma;<sub>n</sub>c<sub>n</sub>n<sup>r</sup>=0 for all r (and converging absolutely). This doesn't involve anything more than solving some linear equations to solve this for any finite set of powers r. Then, by keeping adding more terms to extend to all positive integers r, you get the infinite sequence c<sub>n</sub>. The two distributions can then be obtained by taking the positive and negative parts of c<sub>n</sub>.</p> http://mathoverflow.net/questions/3525/when-are-probability-distributions-completely-determined-by-their-moments/3886#3886 Answer by Ori Gurel-Gurevich for When are probability distributions completely determined by their moments? Ori Gurel-Gurevich 2009-11-03T01:24:56Z 2009-11-03T01:24:56Z <p>Regarding your second question:</p> <p>The Renyi entropy depends only on the probabilities, and not on the values the RV take; any 1-1 function of the RV have the same entropy.</p> <p>If you's asking whether the Renyi entropy determines the sequence of probabilities p<sub>i</sub>, then the answer is yes. Assume WLOG that p<sub>i</sub> are in descending order. Then the limit when a tends to infinity, of H<sub>a</sub> is p<sub>0</sub>. Once you know p<sub>0</sub>, it is easy to calculate the entropy for the sequence p<sub>1</sub>, p<sub>2</sub>,.. which then allows us to find p<sub>1</sub>, etc.</p> http://mathoverflow.net/questions/3525/when-are-probability-distributions-completely-determined-by-their-moments/4787#4787 Answer by Steve Flammia for When are probability distributions completely determined by their moments? Steve Flammia 2009-11-09T23:49:12Z 2009-11-09T23:49:12Z <p>Thinking about the Renyi part of this question again today, I realized that there is a simple and elegant way to show the equivalence of knowing the Renyi entropies and knowing the probabilities (in principle) without taking limits. See <a href="http://mathoverflow.net/questions/3525/when-are-probability-distributions-completely-determined-by-their-moments/3886#3886" rel="nofollow">Ori's comments</a>, also.</p> <p>Suppose we have just a finite number outcomes. Then we can place all of the probabilities for each outcome on the diagonal of a large matrix. The Renyi entropies are basically just the traces of the powers of this matrix for integer values of $\alpha$. We would like to show that knowing these trace powers is equivalent to knowing the probabilities themselves. Intuitively, this seems clear, since it is just an overdetermined system of polynomial equations, but <em>a priori</em> it isn't clear that there isn't some weird degeneracy hidden somewhere that would preclude a unique solution. So, we have the trace powers, and as a function of the probabilities, these are just the power sums. We can use the <a href="http://en.wikipedia.org/wiki/Newton%27s%5Fidentities" rel="nofollow">Newton-Girard identities</a> to transform these into the elementary symmetric polynomials. Then we can express the characteristic polynomial of our large matrix as a <a href="http://en.wikipedia.org/wiki/Newton%27s%5Fidentities#Application%5Fto%5Fthe%5Fcharacteristic%5Fpolynomial%5Fof%5Fa%5Fmatrix" rel="nofollow">sum over these</a>. The roots of this polynomial are of course the eigenvalues, which are just the probabilities in question. </p>