1
$\begingroup$

Let $P,Q,R$ be probability measures on the real line. If these are discrete, we can show \begin{equation} D_{\mathrm{KL}}(P\ast R\,\Vert\,Q\ast R)\le D_{\mathrm{KL}}(P\,\Vert\,Q) \end{equation} by using the log sum inequality, where $D_{\mathrm{KL}}(P\,\Vert\,Q)=\int \log\left(\frac{dP}{dQ}\right) dP$ is KL divergence and $\ast$ is convolution. Is the above inequality true when $P,Q,R$ are not necessarily discrete?

$\endgroup$
  • $\begingroup$ I am specially interested in a case where R is normal distribution. $\endgroup$ – selami Feb 12 '19 at 7:05
1
$\begingroup$

The KL divergence cannot increase after passing both distributions through the same Markov kernel (in your case, convolution with $R$). This is an immediate consequence of the data processing inequality: https://en.wikipedia.org/wiki/Data_processing_inequality

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.