4
$\begingroup$

For continuous distributions on x>0 with known mean m, the exponential distribution f(x) = (1/m)exp(-x/m) is the maximum entropy distribution, with entropy H(f) = ln(m)+1. I have a problem where I know the P-th quantile Q and I want to know the maximum entropy distribution with that quantile.

The exponential distribution with P-th quantile Q has mean m = Q/ln(1-P). As stated this is the maximum entropy distribution for all distributions with mean m. Is it also the maximum entropy distribution over all continuous distributions with P-th quantile Q? If not, what is the maximum entropy distribution. Any help greatly appreciated.

Thanks,

Trevor

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

The quantile alone is insufficient to define a maximum entropy density. Intuitively this is because the quantile is a single point and is not enough to prescribe an entire density; you must specify additional moments.

A related fact is that quantiles are not sufficient statistics for any distributions on $\mathbb{R}$, as noted here on page 17.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks. Let me ask a bit differently. The context is that an expert will articulate a prior in terms such as, "I am 50% confident that total error does not exceed $100,000." We can usually assume mode zero. It is not unreasonable to assume that the expert's implicit prior over the entire range of potential error is an exponential with 50th percentile = 100,000 (and therefore mean = 100,000 / 0.7). Does this have higher entropy than other probability distributions one might possibly assume with the same 50th percentile? $\endgroup$
    – Trevor Stewart
    Commented Mar 21, 2012 at 16:12
  • $\begingroup$ The obstacle to defining a maximum entropy distribution on the real numbers using only a finite number of quantiles is the unbounded support. If you can reasonably supply an upper and a lower bound, then the maximum entropy distribution with a certain fixed quantiles is well defined and is the uniform distribution weighted over subintervals in order to satisfy the quantile locations. An alternative approach that I favor is to jettison the entropy metric and instead interpolate a smooth cdf that runs through your prescribed quantiles. This can be done using piecewise Hermite interpolating $\endgroup$
    – R Hahn
    Commented Mar 21, 2012 at 17:00
  • $\begingroup$ Okay, but it is given that the exponential distribution with mean m is an appropriate representation of the expert's prior (where m is calculated from the known quantile Q). We know that this is the maximum entropy distribution for the class of continuous distributions on X > 0 with mean m. I now want to consider the class of all continuous distributions on x > 0 with quantile Q. Are you saying that there might be distributions in that class with higher entropy than my exponential with mean m? I appreciate your patience--I am dealing with entropy for the first time! $\endgroup$
    – Trevor Stewart
    Commented Mar 21, 2012 at 18:49
  • $\begingroup$ Now I get it... finally. Thanks. An easy counter-example is a gamma distribution with shape = 1/2 and Q the 50th percentile. It has greater entropy than the exponential (gamma(1)) with 50th percentile Q. $\endgroup$
    – Trevor Stewart
    Commented Mar 24, 2012 at 14:12
  • $\begingroup$ Sorry, I didn't see your earlier comment. Glad you've figured it out. The problem is that "the class of all continuous distributions on x > 0 with quantile Q" does not admit a well-defined maximum entropy distribution at all. $\endgroup$
    – R Hahn
    Commented Mar 24, 2012 at 19:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .