1
$\begingroup$

Suppose there are two continuous distributions whose pdfs are $p_1$ and $p_2$, defined on a common support $\mathcal{X}$. Suppose that there is a conditional pdf (the channel) $M:\mathcal{X}\times \mathcal{Y}\rightarrow \mathbb{R}$. Two pdfs $q_1$ and $q_2$ are generated from $p_1$ and $p_2$ by passing $M$. To be more precise,

$q_1(y)=\int p_1(x)M(y|x) dx$,

$q_2(y)=\int p_2(x)M(y|x) dx$.

My question is: Does the following inequality

$D(q_1||q_2)\leq D(p_1||p_2)$

always hold? $D$ denotes the KL divergence.

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ Yes, this is the data processing inequality. $\endgroup$
    – Alf
    Commented Dec 20, 2023 at 3:37
  • 2
    $\begingroup$ I don't immediately see how to apply the data processing inequality here, but it sure feels similar. jkds, can you say if something goes wrong with a direct proof (plugging in for $q_1,q_2$)? $\endgroup$
    – usul
    Commented Dec 20, 2023 at 3:50
  • $\begingroup$ @usul The data processing inequality for mutual information is much easier to prove. However, now we need to prove the inequality for KL divergence. $\endgroup$
    – jkfds
    Commented Dec 21, 2023 at 4:56
  • $\begingroup$ stat.yale.edu/~yw562/teaching/598/lec04.pdf The proof is here. Thanks. $\endgroup$
    – jkfds
    Commented Dec 21, 2023 at 4:57
  • $\begingroup$ @IosifPinelis Yes. This is indeed the data processing inequality. $\endgroup$
    – jkfds
    Commented Dec 25, 2023 at 2:46

1 Answer 1

Reset to default
1
$\begingroup$

As Alf said, this is the data processing inequality -- see e.g. Theorem 7.2.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I have checked. Yes, it is data processing inequality. Thanks $\endgroup$
    – jkfds
    Commented Dec 25, 2023 at 2:46
  • $\begingroup$ @jkfds : All right. If you are satisfied with the answer, then these guidelines may be relevant here. $\endgroup$
    – Iosif Pinelis
    Commented Dec 25, 2023 at 2:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .