0
$\begingroup$

This question concerns distributions $\mu$ over the naturals $\mathbb{N}=\{1,2,\ldots\}$. For $q\ge1$, let us define the $q$th moment of entropy: $$ H_q(\mu)=\sum_{i=1}^\infty \mu(i)|\log\mu(i)|^q, $$ so $H_1(\mu)$ is just the usual entropy.

I am interested in a sequence of distributions $\mu_n$ satisfying the following properties:

  1. $\mu_n(1)\to1$ as $n\to\infty$
  2. $\limsup_{n\to\infty}H_2(\mu_n)<\infty$
  3. $\liminf_{n\to\infty}H_1(\mu_n)>0$

Does such a sequence exist? I'd be satisfied with mere existence (though a construction would, of course, be nice).

If such a sequence does not exist (proof?), I'd be happy with the following weakening:

1'. $\mu_n(1)\to1$ at some fixed rate

2'. same as 2

3'. $H_1(\mu_n)\to0$ at an arbitrarily slow rate.

$\endgroup$

1 Answer 1

2
$\begingroup$

Suppose that $\mu_n(1)=1-t_n$, $\mu_n(2)=\cdots=\mu_n(n+1)=t_n/n$, and $\mu_n(n+2)=\mu_n(n+3)=\cdots=0$, where $t_n:=1/\ln n$ and $n\ge3$. Then $\mu_n(1)\to1$ and $H_1(\mu_n)\to1$. So, 1' and 2' hold; one may say 2' holds with an infinitely slow rate. It is easy to modify this example to have 2' hold with an arbitrarily slow rate.


After the editing of the question, the answer becomes no. Indeed, suppose that $\mu_n(1)=1-t_n$, where $t_n\downarrow0$, and suppose that $H_1(\mu_n)\to0$ so slowly that $$H_1(\mu_n)\ge1\Big/\sqrt{\ln\frac1{t_n}}$$ eventually (i.e., for all large enough $n$).

Then $\mu_n(j)\le t_n$ for $j\ge2$ and therefore $$\ln^2\frac1{\mu_n(j)}\ge\ln\frac1{t_n}\;\ln\frac1{\mu_n(j)},$$ whence eventually $$\begin{aligned} H_2(\mu_n)&\ge\sum_{j\ge2}\mu_n(j)\ln^2\frac1{\mu_n(j)} \\ &\ge\sum_{j\ge2}\mu_n(j)\ln\frac1{\mu_n(j)}\; \ln\frac1{t_n} \\ &=\Big(H_1(\mu_n)-(1-t_n)\ln\frac1{1-t_n}\Big)\ln\frac1{t_n} \\ &\ge \Big(1\Big/\sqrt{\ln\frac1{t_n}}(1-o(1))\Big)\ln\frac1{t_n}\to\infty. \end{aligned} $$

Thus, whenever 1' and 3' hold, 2' cannot hold.

$\endgroup$
5
  • $\begingroup$ But what about $H_2(\mu_n)$? $\endgroup$ Commented Jan 31, 2021 at 16:15
  • $\begingroup$ Ah, I see -- my "weakening" was ambiguous. Fixed. $\endgroup$ Commented Jan 31, 2021 at 16:16
  • $\begingroup$ I now have reason to believe that even the "weakening" is impossible. $\endgroup$ Commented Jan 31, 2021 at 16:18
  • $\begingroup$ Thank you! I had concluded this from much more roundabout reasons -- but your argument is much slicker! $\endgroup$ Commented Jan 31, 2021 at 17:10
  • $\begingroup$ FWIW: my cumbersome, roundabout proof follows from Theorem D of Foster-Wyner: deanfoster.net/research/low_limits_entropy.pdf (by contradiction: if a sequence satisfying either of my set of conditions existed, it would be a counter-example to their result). $\endgroup$ Commented Jan 31, 2021 at 17:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .