5
$\begingroup$

Define the (differential) entropy for a density $f$ as

$$ H(f) :=-\int_{0}^{1} f(x) \log_{2}(f(x)) dx \, .$$

I am trying to find a $f \in L_{1}([0,1])$ such that $f\geq 0, \int_{0}^{1} f(x) dx = 1$ and $H(f)= -\infty $. I am not sure if this is possible? Note that as $ f\in L_{1}([0,1])$, the measure induced by $f$ must be absolutely continuous with respect to Lebesgue measure.

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ Try $f(x) = (x(1+\log x)^2)^{-1}$ $\endgroup$
    – Deane Yang
    Commented Jul 8, 2018 at 4:29
  • $\begingroup$ Thanks. However, $f \notin L_{1}([0,1])$. $\int_{0}^{1} f = -(1+log(x))^{-1}|_{0}^{1}= \infty$, right? $\endgroup$
    – Tejas Bhojraj
    Commented Jul 8, 2018 at 5:11
  • $\begingroup$ Sorry, it has to be $1-\log x$ and not $1 + \log x$, if it works at all. $\endgroup$
    – Deane Yang
    Commented Jul 8, 2018 at 5:14
  • $\begingroup$ Yes, you are right; with $1-log(x)$, atleast $f \in L_{1}([0,1])$. $\int_{0}^{1} f = (1-log(x))^{-1}|_{0}^{1}= 1$. I still need to work out $H(f)$. And my previous comment was wrong, sorry! $\endgroup$
    – Tejas Bhojraj
    Commented Jul 8, 2018 at 5:30
  • 2
    $\begingroup$ Quick commentary on how to find this example (using natural log instead of log base 2): Note that $\|f\|_p$ is an increasing function of $p$ and its limit as $p\rightarrow 0$ is $\exp -H(f)$. So $f$ must be in $L^1$ but not $L^p$ for any $p > 1$. $f = x^{-\alpha}$, $\alpha > 0$, does not work, so the next thing to try is $x^{-1}(1-\log x)^{-\beta}$, where we add 1 to $-\log x$ to prevent the function from blowing up at $x = 1$. This blows up at $0$ slower than $x^{-1}$ but faster than $x^{-\alpha}$ for any $\alpha < 1$. Testing this, it works if $1 < \beta \le 2$. $\endgroup$
    – Deane Yang
    Commented Jul 8, 2018 at 16:32

2 Answers 2

Reset to default
7
$\begingroup$

For $x\in(0,1)$, let \begin{equation} f(x):=\frac1{x\ln^2\frac ex}. \tag{1} \end{equation} Then $f\ge0$ and $\int_0^1f(x)\,dx=1$. On the other hand, $\ln f(x)\sim\ln\frac ex$ as $x\downarrow0$. So, $\ln f(x)\ge\frac12\ln\frac ex$ for some $c\in(0,1)$ and all $x\in(0,c)$. So, $$\int_0^1 f(x)\ln f(x)\,dx\ge \int_0^c \frac1{x\ln^2\frac ex}\frac12\ln\frac ex\,dx=\infty,$$ as desired.

A few words on how this example was found. First, by Jensen's inequality and the condition $\int f=1$, we have $\int f\ln f\ge\int f\,\ln\int f=0$, so that the minimum of $\int f\ln f$ given $\int f=1$ is attained when $f=1$. So, to get a large value of $\int f\ln f$, it makes sense to try letting $f$ be as non-uniform (non-constant) as possible. Taking then $f=f_a:=a\,1_{[0,1/a]}$ with $a\to\infty$ (so that $f$ "explodes" at $0$ to $\infty$), we have $\int f=1$ and $\int f\ln f=\ln a\to\infty$. Now, instead of a family $(f_a)$ of functions exploding to $\infty$ at $0$, we can try to find a single function exploding to $\infty$ at $0$ as fast as possible, but so that the condition $\int f=1$ holds. This results in the function $f$ in (1), which happens to be the same as the one suggested by Deane Yang.

Improve this answer
$\endgroup$
1
$\begingroup$

Let $f$ be a piecewise constant function which takes value $f_i$ on an interval of length $p_i$, and let $q_i=f_i p_i$. Then $$ H(f) = - \int f \log f\,dx = - \sum q_i \log\frac{q_i}{p_i} = - D_{KL} (Q||P) $$ is the Kullback--Leibler divergence of of the distribution $P=(p_i)$ from the distribution $Q=(q_i)$. So, one just has to exhibit a pair of discrete distributions with infinite KL divergence.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .