Skip to main content
1 of 4
Bjørn Kjos-Hanssen
  • 24.8k
  • 3
  • 58
  • 114

I'll show that Yes, if you rule out an event of sufficiently small probability then the entropy decreases.

I'll change the notation around a bit so that my $p_1,\dots,p_n$ correspond to your $q_{n+1},\dots,q_1$.

Suppose $\sum_{i=1}^n p_i=1$ and we are given that $p_1$ did not occur. Then the new probability of $p_k$, $k\ne 1$, is $$ \hat p_k = \frac{p_k}{1-p_1}. $$ So the new entropy $H(\hat p)$ is (where $S(p)=-\log_2(p)$ and $q$ is the experiment that only determines whether $p_1$ occurs) $$ \sum_{k=2}^n S\left(\frac{p_k}{1-p_1}\right)\frac{p_k}{1-p_1} = \frac1{1-p_1}\sum_{k=2}^n [S(p_k)-S(1-p_1)]{p_k} $$ $$ = \frac1{1-p_1}\left(\sum_{k=2}^n S(p_k)p_k - (1-p_1)S(1-p_1)\right) = \frac1{1-p_1}\left(H(p) - S(p_1)p_1 - (1-p_1)S(1-p_1)\right) $$ $$ = \frac1{1-p_1}\left(H(p) - H(q)\right) \le H(p)\quad\text{(as we want) iff } $$ $$ \frac{H(q)}{p_1}\ge H(p). $$ The function $p_1\mapsto H(q)/p_1$ is $$ p_1\mapsto -\log_2(p_1) - \frac{1-p_1}{p_1}\log_2(1-p_1) $$ which goes to infinity as $p_1\rightarrow 0$, whereas $H(p)$ barely changes (and is in any case bounded by 1) as $p_1\rightarrow 0$, so it is certainly enough to take $p_1$ very small.

Bjørn Kjos-Hanssen
  • 24.8k
  • 3
  • 58
  • 114