4
$\begingroup$

Define the (differential) entropy for a density $f$ as

$$ H(f) :=-\int_{0}^{1} f(x) \log_{2}(f(x)) dx \, .$$

I am trying to find a $f \in L_{1}([0,1])$ such that $f\geq 0, \int_{0}^{1} f(x) dx = 1$ and $H(f)= -\infty $. I am not sure if this is possible? Note that as $ f\in L_{1}([0,1])$, the measure induced by $f$ must be absolutely continuous with respect to Lebesgue measure.

$\endgroup$
6
  • 1
    $\begingroup$ Try $f(x) = (x(1+\log x)^2)^{-1}$ $\endgroup$
    – Deane Yang
    Jul 8 '18 at 4:29
  • $\begingroup$ Thanks. However, $f \notin L_{1}([0,1])$. $\int_{0}^{1} f = -(1+log(x))^{-1}|_{0}^{1}= \infty$, right? $\endgroup$ Jul 8 '18 at 5:11
  • $\begingroup$ Sorry, it has to be $1-\log x$ and not $1 + \log x$, if it works at all. $\endgroup$
    – Deane Yang
    Jul 8 '18 at 5:14
  • $\begingroup$ Yes, you are right; with $1-log(x)$, atleast $f \in L_{1}([0,1])$. $\int_{0}^{1} f = (1-log(x))^{-1}|_{0}^{1}= 1$. I still need to work out $H(f)$. And my previous comment was wrong, sorry! $\endgroup$ Jul 8 '18 at 5:30
  • 2
    $\begingroup$ Quick commentary on how to find this example (using natural log instead of log base 2): Note that $\|f\|_p$ is an increasing function of $p$ and its limit as $p\rightarrow 0$ is $\exp -H(f)$. So $f$ must be in $L^1$ but not $L^p$ for any $p > 1$. $f = x^{-\alpha}$, $\alpha > 0$, does not work, so the next thing to try is $x^{-1}(1-\log x)^{-\beta}$, where we add 1 to $-\log x$ to prevent the function from blowing up at $x = 1$. This blows up at $0$ slower than $x^{-1}$ but faster than $x^{-\alpha}$ for any $\alpha < 1$. Testing this, it works if $1 < \beta \le 2$. $\endgroup$
    – Deane Yang
    Jul 8 '18 at 16:32
6
$\begingroup$

For $x\in(0,1)$, let \begin{equation} f(x):=\frac1{x\ln^2\frac ex}. \tag{1} \end{equation} Then $f\ge0$ and $\int_0^1f(x)\,dx=1$. On the other hand, $\ln f(x)\sim\ln\frac ex$ as $x\downarrow0$. So, $\ln f(x)\ge\frac12\ln\frac ex$ for some $c\in(0,1)$ and all $x\in(0,c)$. So, $$\int_0^1 f(x)\ln f(x)\,dx\ge \int_0^c \frac1{x\ln^2\frac ex}\frac12\ln\frac ex\,dx=\infty,$$ as desired.


A few words on how this example was found. First, by Jensen's inequality and the condition $\int f=1$, we have $\int f\ln f\ge\int f\,\ln\int f=0$, so that the minimum of $\int f\ln f$ given $\int f=1$ is attained when $f=1$. So, to get a large value of $\int f\ln f$, it makes sense to try letting $f$ be as non-uniform (non-constant) as possible. Taking then $f=f_a:=a\,1_{[0,a]}$ with $a\to\infty$ (so that $f$ "explodes" at $0$ to $\infty$), we have $\int f=1$ and $\int f\ln f=\ln a\to\infty$. Now, instead of a family $(f_a)$ of functions exploding to $\infty$ at $0$, we can try to find a single function exploding to $\infty$ at $0$ as fast as possible, but so that the condition $\int f=1$ holds. This results in the function $f$ in (1), which happens to be the same as the one suggested by Deane Yang.

$\endgroup$
1
$\begingroup$

Let $f$ be a piecewise constant function which takes value $f_i$ on an interval of length $p_i$, and let $q_i=f_i p_i$. Then $$ H(f) = - \int f \log f\,dx = - \sum q_i \log\frac{q_i}{p_i} = - D_{KL} (Q||P) $$ is the Kullback--Leibler divergence of of the distribution $P=(p_i)$ from the distribution $Q=(q_i)$. So, one just has to exhibit a pair of discrete distributions with infinite KL divergence.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.