-1
$\begingroup$

For $X$ a measurable space and $P,Q$ two probability measures on $X$ s.t. $Q$ is absolutely continuous with respect to $P$, the relative entropy is defined as $$D(Q\|P)=\int_X \log(\frac{dQ}{dP})dQ,$$ where $\frac{dQ}{dP}$ is a Radon-Nikodym derivative of $Q$ with respect to $P$. Now, the question is,

What is the sufficient condition for $D(Q\|P)=0$ when $D(Q\|P)$ in defined in infinite probability space?

By infinite probability space I mean "sample space, $\Omega$, which is the set of all possible outcomes, is not finite."

Improve this question
$\endgroup$
4
  • $\begingroup$ As Michele shows in his answer the divergence is zero if and only if the two probabilities are equal. This is an elementary property of KL-Divergence (in fact Michele gave the full proof). Have a look at the wikipedia page and any book on the subject (e.g. Gray's book on information theory): en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence $\endgroup$
    – Pablo Lessa
    Commented Dec 5, 2014 at 12:22
  • $\begingroup$ Dear Pablo, According to the link which you have sent, there Gibb's inequality is used and in Gibb's inequality they have used that finite point in sample space. That is my problem, if it is infinite then what is the condition. $\endgroup$
    – Sedi
    Commented Dec 5, 2014 at 13:15
  • $\begingroup$ Sedi: There's no real issue with the probability space being infinite you just need to study up a bit. The proof Michele gives below works verbatim. Jensen's inequality is valid for general probability measures. Check out the wikipedia page for Jensen's inequality or any book on information theory (e.g. Gray's book Chapter 7, lemma 7.1). I don't think this question is appropriate for mathoverflow since it's treated in detail in practically any reference on the subject. Best of luck. $\endgroup$
    – Pablo Lessa
    Commented Dec 5, 2014 at 13:29
  • $\begingroup$ Dear Pablo, Thank you for your wishes. All your references are in finite space. I haven't seen in general form. Anyways, its maybe correct, I need to study more. $\endgroup$
    – Sedi
    Commented Dec 5, 2014 at 14:32

1 Answer 1

Reset to default
2
$\begingroup$

This should be just a comment (I don't have enough points). Using Jensen's inequality, with the equality case, you obtain that $\log \frac{dQ}{dP}$ must be constant a.e., so equal to $0$ a.e.

Improve this answer
$\endgroup$
5
  • $\begingroup$ What do you mean by partial answer? What is equal to 0 a.s. in the end? $\endgroup$
    – Sedi
    Commented Dec 5, 2014 at 12:00
  • $\begingroup$ I mean that I should add some more detail $\endgroup$
    – user47274
    Commented Dec 5, 2014 at 12:02
  • $\begingroup$ Dear Michele, to me, it seems for your proof, we need decomposition of $\log(\frac{dQ}{dP})\frac{dQ}{dP}$ which is not concave any more... Is not it? $\endgroup$
    – Sedi
    Commented Dec 5, 2014 at 13:17
  • $\begingroup$ I'm sorry Sedi, but now you have changed the integral with respect to dP into dQ! I think that with what Pablo told you, you will be able to find a proof on the Internet. $\endgroup$
    – user47274
    Commented Dec 5, 2014 at 20:08
  • $\begingroup$ Thank you Michele, I think I got the thing which I were looking for after discussion here. Thank you $\endgroup$
    – Sedi
    Commented Dec 6, 2014 at 5:23

Not the answer you're looking for? Browse other questions tagged .