5
$\begingroup$

Suppose we have a continuous probability distribution with density function $f$ whose support is $[a,b]$ and we know that for some finite set of values $\{ v_i \}_{i=1}^n$ between $a$ and $b$ that the $\operatorname{CDF}[f,v_i]=q_i$ where $$ a < v_1 \le \cdots \le v_n < b $$ Essentially, we know the CDF of the distribution at various points in the interval, but we don't know it over the entire interval $[a,b]$.

How do we figure out the maximum entropy probability distribution that satisfies these constraints?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Let $X \sim f$ which implies that $X \in [a,b]$ almost surely. In terms of the CDF of $X$ at the points $\{ v_i \}_{i=1}^n$, define $$ p_i := \mathbb{P}( v_{i-1} < X \le v_i ) = q_i - q_{i-1} \;, \quad 1 \le i \le n+1 $$ where we have introduced $v_0 = a$, $v_{n+1} = b$, $q_{0}=0$ and $q_{n+1} = 1$. Note that $\sum_{1 \le i \le n+1} p_i = 1$ since the sum telescopes.

More to the point, we wish to maximize the entropy $h(f)$ subject to the constraints that: $\int_a^b f(x) dx = 1$ and $\int_{v_{i-1}}^{v_i} f(x) dx = p_i$ for $1 \le i \le n+1$. As discussed in the reference below, the density of the maximum entropy distribution which satisfies these constraints is given by: $$ f(x) = Z^{-1} \exp\left( \sum_{i=1}^{n+1} \lambda_i 1_{(v_{i-1}, v_i]}(x) \right) 1_{[a,b]}(x) $$ where $Z$ is a normalization constant chosen such that $\int_a^b f(x) =1$ and where $\{ \lambda_i \}_{i=1}^{n+1}$ are Lagrange multipliers chosen such that $\int_{v_i}^{v_{i+1}} f(x) dx = p_i$. Eliminating these Lagrange multipliers and writing the density in terms of the given quantiles yields $$ f(x) = \sum_i\frac{q_i - q_{i-1}}{v_i - v_{i-1}} 1_{(v_{i-1}, v_i]}(x) \;. $$

Reference

Cover, T. M., and J. A. Thomas. "Chapter 12, Maximum Entropy." Elements of Information Theory. John Wiley & Sons, 2012.

Improve this answer
$\endgroup$
4
  • $\begingroup$ The notation $1_[x,y]$ means a function whose value is 1 on the interval [x,y] and 0 elsewhere? $\endgroup$
    – mathlawguy
    Commented Oct 28, 2016 at 19:07
  • $\begingroup$ Thanks for the help. I'm a little confused by your answer; it is probably the result of my not being as expert in math as desirable. You appear to say at first that the maximum entropy distribution is some sort of exponential distribution in which the parameter depends on which part of the domain we are in. But then you write "Eliminating these Lagrange multipliers and writing the density in terms of the quantiles yields ... and then you have what looks to me to be a piecewise uniform distribution. How did we get from an exponential distribution to a uniform distribution? $\endgroup$
    – mathlawguy
    Commented Oct 28, 2016 at 19:43
  • $\begingroup$ (i) Yes, that is exactly what the notation means; sorry for not defining it. (ii) The way to eliminate the $i$th Lagrange multiplier from $f(x)$ is to apply the $i$th constraint $\int_{v_{i-1}}^{v_i} f(x) dx = p_i$ which implies that $Z^{-1} \exp(\lambda_i) (v_i - v_{i-1}) = p_i$. Then use this last equation to eliminate $\exp(\lambda_i)$ from $f(x)$ to obtain the final equation given in the answer. $\endgroup$
    – Nawaf Bou-Rabee
    Commented Oct 28, 2016 at 20:14
  • $\begingroup$ Did you mean to write $f(x) = \sum_i \ldots$ in your last equation? $\endgroup$
    – user76284
    Commented Jul 8, 2021 at 7:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .