For continuous distributions on x>0 with known mean m, the exponential distribution f(x) = (1/m)exp(-x/m) is the maximum entropy distribution, with entropy H(f) = ln(m)+1. I have a problem where I know the P-th quantile Q and I want to know the maximum entropy distribution with that quantile.

The exponential distribution with P-th quantile Q has mean m = Q/ln(1-P). As stated this is the maximum entropy distribution for all distributions with mean m. Is it also the maximum entropy distribution over all continuous distributions with P-th quantile Q? If not, what is the maximum entropy distribution. Any help greatly appreciated.




The quantile alone is insufficient to define a maximum entropy density. Intuitively this is because the quantile is a single point and is not enough to prescribe an entire density; you must specify additional moments.

A related fact is that quantiles are not sufficient statistics for any distributions on $\mathbb{R}$, as noted here on page 17.

  • $\begingroup$ Thanks. Let me ask a bit differently. The context is that an expert will articulate a prior in terms such as, "I am 50% confident that total error does not exceed $100,000." We can usually assume mode zero. It is not unreasonable to assume that the expert's implicit prior over the entire range of potential error is an exponential with 50th percentile = 100,000 (and therefore mean = 100,000 / 0.7). Does this have higher entropy than other probability distributions one might possibly assume with the same 50th percentile? $\endgroup$ – Trevor Stewart Mar 21 '12 at 16:12
  • $\begingroup$ The obstacle to defining a maximum entropy distribution on the real numbers using only a finite number of quantiles is the unbounded support. If you can reasonably supply an upper and a lower bound, then the maximum entropy distribution with a certain fixed quantiles is well defined and is the uniform distribution weighted over subintervals in order to satisfy the quantile locations. An alternative approach that I favor is to jettison the entropy metric and instead interpolate a smooth cdf that runs through your prescribed quantiles. This can be done using piecewise Hermite interpolating $\endgroup$ – R Hahn Mar 21 '12 at 17:00
  • $\begingroup$ Okay, but it is given that the exponential distribution with mean m is an appropriate representation of the expert's prior (where m is calculated from the known quantile Q). We know that this is the maximum entropy distribution for the class of continuous distributions on X > 0 with mean m. I now want to consider the class of all continuous distributions on x > 0 with quantile Q. Are you saying that there might be distributions in that class with higher entropy than my exponential with mean m? I appreciate your patience--I am dealing with entropy for the first time! $\endgroup$ – Trevor Stewart Mar 21 '12 at 18:49
  • $\begingroup$ Now I get it... finally. Thanks. An easy counter-example is a gamma distribution with shape = 1/2 and Q the 50th percentile. It has greater entropy than the exponential (gamma(1)) with 50th percentile Q. $\endgroup$ – Trevor Stewart Mar 24 '12 at 14:12
  • $\begingroup$ Sorry, I didn't see your earlier comment. Glad you've figured it out. The problem is that "the class of all continuous distributions on x > 0 with quantile Q" does not admit a well-defined maximum entropy distribution at all. $\endgroup$ – R Hahn Mar 24 '12 at 19:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.