Say the KL divergence between two distributions $A$ and $B$ is $\varepsilon$. Can we give bounds, or a precise computation, of the KL divergence between $A^k$ and $B^k$ (the product distributions)?
2
-
4$\begingroup$ I think the "chain rule" for KL-divergence immediately gives $k\epsilon$, using independence of the $j$th component for $j=1,\dots,k$. $\endgroup$– usulCommented Feb 5, 2018 at 23:13
-
2$\begingroup$ Ah right, that follows directly from the chain rule. For future readers, this is Theorem 5.3 of this PDF, and applying it to the product distribution is straightforward. Thanks! $\endgroup$– TedCommented Feb 5, 2018 at 23:28
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Transforming usul's comment into a proper answer: if the KL divergence between $A$ and $B$ is $\varepsilon$, the KL divergence between $A^k$ and $B^k$ is $k\varepsilon$. This follows directly from the chain rule (Theorem 5.3 of this PDF, applied to a situation where $x$ and $y$ are independent).