Skip to main content
1 of 5

Let $X \sim f$ which implies that $X \in [a,b]$ almost surely. In terms of the CDF of $X$ at the points $\{ v_i \}_{i=1}^n$, define $$ p_i := \mathbb{P}( v_{i-1} < X \le v_i ) = q_i - q_{i-1} \;, \quad 1 \le i \le n+1 $$ where we have introduced $v_0 = a$, $v_{n+1} = b$, $q_{0}=0$ and $q_{n+1} = 1$. Note that $\sum_{i=1}^{n+1} p_i = 1$ since the sum telescopes.

Consider the class of distributions whose density can be written as: $$ f(x) = p_i 1_{(v_{i-1}, v_i]}(x) $$ For this class of probability distributions, their differential entropy reduces to a weighted sum: $$ - \int_a^b f(x) \log(f(x)) dx = - \sum_{1 \le i \le n+1} p_i \log(p_i) (v_{i} - v_{i-1}) $$ This weighted sum is a concave function of its input vector $(p_1, \cdots, p_{n+1})$, and by using a Lagrange multiplier, one can show that it attains its maximum at: $$ p_i = Z^{-1} \exp( - (v_i - v_{i-1}) ) \;, \quad 1 \le i \le n+1 $$ where $Z$ comes from eliminating the Lagrange multiplier, and is a normalization constant chosen such that: $\sum_{1 \le i \le n+1} p_i =1 $. In the special case where $q_i - q_{i-1} = h > 0$ for $1 \le i \le n+1$, then $Z=\exp(-h) (n+1)$ and $p_i = 1/(n+1)$ for $1 \le i \le n+1$.