Skip to main content
1 of 3
Algernon
  • 1.8k
  • 1
  • 17
  • 17

Claim. If $\|P-Q\|\leq\varepsilon\leq\frac{1}{2}$, then $|H(P)-H(Q)| \leq H(\varepsilon) + \varepsilon\log N$.

Proof. Let $\varepsilon':=\|P-Q\|$. Let $(X,Y)$ be an optimal coupling of $P$ and $Q$, so that \begin{align} \mathbb{P}(X\neq Y) = \|P-Q\| \;. \end{align} Using a standard construction, we can assume that $X$ and $Y$ have the particular form \begin{align} X &:= \begin{cases} Z & \text{if $B=0$,} \\ \tilde{X} & \text{if $B=1$,} \end{cases} & Y &:= \begin{cases} Z & \text{if $B=0$,} \\ \tilde{Y} & \text{if $B=1$,} \end{cases} \end{align} where $B$, $Z$ and $(\tilde{X},\tilde{Y})$ are independent and $B\sim\text{Bern}(\varepsilon')$.

Note that \begin{align} H(X|B) \leq H(X) \leq H(B) + H(X|B) \;. \end{align} For $H(X|B)$ we can write \begin{align} H(X|B) &= \varepsilon' H(X|B=1) + (1-\varepsilon') H(X|B=0) \\ &= \varepsilon' H(\tilde{X}) + (1-\varepsilon') H(Z) \;. \end{align} Thus, \begin{align} \varepsilon' H(\tilde{X}) + (1-\varepsilon') H(Z) &\leq H(X) \leq H(B) + \varepsilon' H(\tilde{X}) + (1-\varepsilon') H(Z) \;, \tag{$\clubsuit$} \end{align} and similarly, \begin{align} \varepsilon' H(\tilde{Y}) + (1-\varepsilon') H(Z) &\leq H(Y) \leq H(B) + \varepsilon' H(\tilde{Y}) + (1-\varepsilon') H(Z) \;. \tag{$\spadesuit$} \end{align}

Combining ($\clubsuit$) and ($\spadesuit$) we get \begin{align} |H(X)-H(Y)| &\leq H(B) + \varepsilon' |H(\tilde{X}) - H(\tilde{Y})| \\ &\leq H(\varepsilon') + \varepsilon' \log N \\ &\leq H(\varepsilon) + \varepsilon \log N \;, \end{align} as claimed. QED

Algernon
  • 1.8k
  • 1
  • 17
  • 17