0
$\begingroup$

Let us say we have two probability measures, $X$ and $Y$ on sample spaces $\Sigma_X$ and $\Sigma_Y$ (which are finite sets, with the largest sigma algebra on each space) and suppose we get measure $Y$ via a function $f: g(X) \rightarrow Y$. What is the Relative Entropy $D_{KL}(X || Y)$ when g is:

  1. An isomorphism
  2. A Surjection, non-isomorphism
  3. An injective non-surgective
$\endgroup$
3
  • $\begingroup$ What is your probability space and what is $g$? The KL divergence is defined for probability measures, not for random variables. $\endgroup$
    – R W
    Dec 5, 2019 at 17:47
  • $\begingroup$ @RW Thanks, for your comment. I will change this for X,Y to be probability measures. Are you suggesting that the relative entropy will depend on the type of function? I will give three questions about different types of functions. $\endgroup$
    – Ben Sprott
    Dec 5, 2019 at 17:51
  • $\begingroup$ What sort of answer are you expecting? This essentially reduces to the question "What is $\log(g(x))$?" - there might be a sensible answer for some specific $g$, like $g(x) = x^n$, but I don't think there's a whole lot to say in general, except maybe some inequalities. $\endgroup$ Dec 5, 2019 at 18:13

1 Answer 1

2
$\begingroup$

As R W noted, the relative entropy depends only on the individual (marginal) distributions of $X$ and of $Y$; it does not depend on the joint distribution of $X$ and $Y$. So, the condition that $Y$ is a function of $X$ is quite irrelevant as far as the relative entropy is concerned, not matter what the function is.

Perhaps you mean the conditional (rather than relative) entropy $$H(Y|X):=-E\ln p(Y|X)$$ of $Y$ given $X$, where $p$ is the conditional probability mass function (pmf) of a random variable $Y$ given $X$ (assuming that the conditional distribution of $Y$ given $X$ is discrete).

If so, then (as you guessed in the original version of your post) $H(Y|X)=0$ if $Y$ is a function of $X$ , because then the conditional pmf $p$ is supported on a singleton set, for each given value of $Y$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.