Question

For each $n \in \mathbb{N}$ let $X_n$ be a random variable taking its values in a finite set $E_n$ with $P(X_n=x_n)>0$ for all $x_n \in E_n$. Say that $X_n$ is asymptotically degenerate if $\min_{x_n \in E_n}\Pr(X_n \neq x_n) \to 0$. Denoting by $h(X)$ the entropy of (the law of) a discrete random variable $X$, assume that $\frac{h(X_n)}{\log k_n}\to0$ with $k_n = \sharp E_n$. Does it imply that $X_n$ is asymptotically degenerate ?

Answer 1

Of course, not. Your "asymptotic degeneracy" condition means that the weight of the maximal atom of the distribution of $X_n$ goes to 1, which is a pretty strong condition. If the sizes of your sets $E_n$ go to infinity, take a sequence $\epsilon_n\to 0$. Then the sequence of measures $\nu_n$ with two atoms of weight $(1-\epsilon_n)/2$ and the remaining mass $\epsilon_n$ equidistributed among the other points of $E_n$ is a counterexample. On the other hand, if the sizes of $E_n$ are uniformly bounded, then $h(X_n)\to 0$ indeed implies asymptotic degeneracy in your sense.