3
$\begingroup$

Say the KL divergence between two distributions $A$ and $B$ is $\varepsilon$. Can we give bounds, or a precise computation, of the KL divergence between $A^k$ and $B^k$ (the product distributions)?

$\endgroup$
  • 4
    $\begingroup$ I think the "chain rule" for KL-divergence immediately gives $k\epsilon$, using independence of the $j$th component for $j=1,\dots,k$. $\endgroup$ – usul Feb 5 '18 at 23:13
  • 2
    $\begingroup$ Ah right, that follows directly from the chain rule. For future readers, this is Theorem 5.3 of this PDF, and applying it to the product distribution is straightforward. Thanks! $\endgroup$ – Ted Feb 5 '18 at 23:28
2
$\begingroup$

Transforming usul's comment into a proper answer: if the KL divergence between $A$ and $B$ is $\varepsilon$, the KL divergence between $A^k$ and $B^k$ is $k\varepsilon$. This follows directly from the chain rule (Theorem 5.3 of this PDF, applied to a situation where $x$ and $y$ are independent).

$\endgroup$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.