# Entropy equals zero?

Imagine you have a shift invariant ($\sigma$-invariant) probability measure $\eta$ in the Bernoulli space $\{0,1\}^{\mathbb{N}}$. Define

$\mathcal{P} = \{,\}$;

$\mathcal{P}^{n} = \mathcal{P}\vee...\vee \sigma^{-n+1}(\mathcal{P})$ (cylinders of length $n$);

$k_{n} = \#\{P \in \mathcal{P}^{n}:\eta(P)\leq\frac{1}{2^{n}}\}$.

For all $n$, enumerate all the cylinders of length $n$ in increasing order with respect the weight that the measure $\eta$ gives to the cylinders. Say $\{P^{n}_{1},...,P^{n}_{2^{n}}\}$. Assume now a simple condition for $\eta$:

$(\sum_{j=1}^{k_{n}}\eta(P^{n}_{j}) +\frac{2^{n}-k_{n}}{2^{n}}) \rightarrow_{n} 0$

Is it true that $h_{\eta}(\sigma)=0$?

Does anyone have any idea about how to prove this? Or, of course, an example the contradicts it?

I could proof that if the entropy is zero, that condition above is satisfied. What I would like to prove is this:

$h_{\eta}(\sigma)=0$ if and only if $(\sum_{j=1}^{k_{n}}\eta(P^{n}_{j}) +\frac{2^{n}-k_{n}}{2^{n}}) \rightarrow_{n} 0$

Thanks for your attention

• As we can see, the measure $\eta$ is giving so much weight to only a "few" cylinders when the $n$ grows. We do not have a sense of regularity here... Its very different to what happens with the Bernoulli measure $(\frac{1}{2},\frac{1}{2})$... – Bruno Brogni Uggioni Oct 6 '15 at 16:01
• I don't see how your statement is valid even for entropy $0$. By the Shannon-McMilan-Breiman theorem, the measure of the bad set (which includes those sets $P_{j}^{n}$ for $j\leq k_n$ for $n\gg 0$) is negligible, as a good set $g$ has big measure, $\mu(g)\geq 2^{-\epsilon n}$, hence we can deal trivially with the LHS of your expression, but for the RHS, just estimating crudely the number of good sets is not enough as you get $2^{\epsilon n}$. So you're assumption is a close cousin of the statement "$k_n$ is rather large, almost as large as possible ~$(1-\epsilon)2^n$", which is not reasonable. – Asaf Oct 7 '15 at 7:08
• Professor Asaf, when you have entropy equals zero, the condition is satisfied. Its a normal computation, but its true... I dont know the other side... – Bruno Brogni Uggioni Oct 7 '15 at 13:27
• Dear Bruno, You are right, I got confused with the directions, as the good sets will be of measure $\geq 2^{-\epsilon n}$, there cannot be many of those (namely, not $C\cdot 2^{n}$ of those sets), hence the RHS goes to zero as-well (this basically estimates the percentage of the good sets). Therefore, the condition you've given is directly implied by the SMB theorem. So you're asking about converse to the SMB theorem, which in your case, Bernoulli shift, seems reasonable. – Asaf Oct 7 '15 at 14:04
• The left hand-side(LHS) is just the sum $\sum\mu(P_{j}^n)$, the right hand side is $(2^n-k_n)/2^n$. Now about SMB, by the SMB theorem, for $\epsilon >0$ and for $n\gg_\epsilon 0$, we can divide the cylinders $P_{i}^{n}$ into two sets - "good" and "bad". The total mass of the bad sets is smaller than $\epsilon$, and the mass of every good cylinder is about $2^{-n(h\pm\epsilon)}$. Assuming $h<1$, you can see that the $k_n$ collection consists of "mainly" bad sets, while there must be $>(1-\epsilon)2^{n(h-\epsilon)}$ good sets (which are outside of the $k_n$ collection). – Asaf Oct 7 '15 at 15:08

## 2 Answers

EDIT - The answer below deals with an ergodic m.p.s

As this question got up-voted, I've decided to fuly write a solution, based on the sketch I've made in the comments.

Fix some $\varepsilon>0$ small, and $n \gg _\varepsilon 0$, and denote by $C_{n}$ to be the cylinders of length $n$. Let $h=h_{\mu}(\sigma)$ be the metric (Kolmogorov-Sinai) entropy of the system $(\Sigma,\sigma,\mu)$. Moreover, assume $\mu$ is $\sigma$-ergodic measure!

By the Shannon-McMillan-Breiman theorem, there exists a partition of $C_n$ into two sets - $G_{n}, B_{n}$ where $G_{n}$ are the ''good'' cylinders, namely for every $g\in G_{n}$ we have $\mu(g_{n})\approx 2^{-n(h\pm\varepsilon)}$. Now for the ''bad'' cylinders, we have that $\sum_{b\in B_{n}}\mu(b) < \varepsilon$.

Moreover, define the set $S_{n}$ to be the ''small'' cylinders, namely $s\in S_{n}$ iff $\mu(s)\leq 2^{-n}$.

From now on, assume $h<1$.

We see that $G_{n} \cap S_{n} = \emptyset$, as $\mu(g)\geq 2^{-n(h+\varepsilon)}>2^{-n}$ for suitably chosen $\varepsilon$ and $g\in G_{n}$, hence $\sum_{s\in S_{n}} \mu(s) \leq \sum_{s\in B_{n}}\mu(s) <\varepsilon$.

Now we want to estimate $2^{n} - |S_{n}|$, as $|G_{n}|\leq 2^{n} - |S_{n}|$, we first bound $|G_{n}|$.

By a crude packing bound we get $|G_{n}|\approx (1-\varepsilon)2^{n(h\pm\varepsilon)}$.

Now we need to estimate $B_{n}\setminus S_{n}$. The atoms in $B_{n}$ are of two types - large atoms (more than a typical one of the atoms in the good set), and small atoms which are not tiny (namely between $2^{-n}$ and $2^{-n(h+\varepsilon)}$). The number of the larger ones is at-most $\varepsilon 2^{n(h+\varepsilon)}$, and the number of the smaller ones is at-most $\varepsilon 2^{n}$, by a simple union bound and again a volume packing argument.

Therefore, the total number of those atoms is $\leq \varepsilon 2^{n}+o(2^{n})$, which translates to $\frac{2^{n}-|S_{n}|}{2^{n}} \lesssim \varepsilon$.

Hence $\lim_{n} \sum_{s\in S_{n}}\mu(s)+\frac{2^{n}-|S_{n}|}{2^{n}} =0$ for any measure with $h<1$.

Notice that for $h=1$ (recall I normalize entropy to $log_{2}$ basis), you get simply the Lebesgue measure, as we have uniqueness of measure of maximal entropy in this system, and in that case, $\sum_{s\in S_{n}}\mu(s)+\frac{2^{n}-|S_{n}|}{2^{n}} =1$ by a simple computation.

• I'm not sure if this a true research question, but I decided to post as the discussion took place in comments, – Asaf Oct 8 '15 at 17:21

I wanted to put this as as comment but I don't have enough reputation for that so I am sorry it is in the wrong place. Anyways, the counting argument described by Asaf can show that if $h_\eta (\sigma)<\log 2$ then in your notation $\sum_{n=1}^{k_n} \eta \left(P_j ^n\right)\to 0$. Indeed, in that case, let $h_\eta (\sigma)<\rho<\log 2$ and denote by $G^n\subset \mathcal{P}^n$ be the collection of all sets of measure greater or equal to $e^{-n\rho}$ (these sets are the good sets in Asaf's reply). By the SMB Theorem $$\eta \left(\cup_{P\in G_n}P\right)\to 1,\ as\ n\to\infty.$$ The conclusion follows since for all $1\leq k\leq k_n$, $P_j ^n\notin G^n$. What I don't see immediately is why entropy $0$ implies $\frac{2^n-k_n}{2^n}\to 0$. It is true that by a counting argument (since $\eta(X)=1$) there are at most $e^{n\rho}=o\left(2^n\right)$ sets in $G^n$ but I don't see why shouldn't there be more than $C2^n$ sets in $\mathcal{P}^n$ of measure greater or equal to $2^{-{(n-1)}}$?

• You should do the packing argument twice, where you pack the bad sets, you're getting $\varepsilon 2^{n}$ bound for free. I suggest reading my full answer. – Asaf Oct 8 '15 at 17:14
• Dear professor @Asaf, thanks for your answer. It surprised me a lot! I thought it was an equivalence, that you needed to have entropy equals zero... but you don't... you only need a shift invariant measure and everything works well... now I will try to understand deeply it... – Bruno Brogni Uggioni Oct 8 '15 at 21:07