Consider continous random variable $X$ with bounded support $\mathcal{X} \subset \mathbb{R}$, discrete random variable $Y$ with finite support $\mathcal{Y}$ , and constants $c_1 ,c_2 \in \mathbb{R}$. Let $p_{XY}(x,y)$ be the probability measure on $\mathcal{X} \times \mathcal{Y}$. Define $H(X,Y)$ and $H(Y\mid X)$ as the joint and conditional entropies. Define $Z = c_1 X_1 + c_2 X_2$ where $X_1$ and $X_2$ are i.i.d. realizations from the probbaility measure on $X$. What are $c_1,c_2$ that maximize $I(Y;Z)$ where $I$ is the mutual information? what about general case with $n$ constants $c_1,...,c_n$?

  • $\begingroup$ You mean $Z$ instead of $RX$, don't you? $\endgroup$ – Jean Duchon May 1 '18 at 10:10
  • $\begingroup$ yes, I have corrected the question. $\endgroup$ – Jeff May 1 '18 at 10:58
  • $\begingroup$ Trivial if you can make the map invertible? Can I choose a $c_1$ and $c_2$ such that you can decode back $X_1$ and $X_2$ from $Z$? $\endgroup$ – Memming May 1 '18 at 13:07
  • $\begingroup$ @Memming the map might not be invertible in general. $\endgroup$ – Jeff May 2 '18 at 7:18
  • $\begingroup$ @MichaelHardy, you are right and that was a typo. I have corrected the question. $\endgroup$ – Jeff May 2 '18 at 7:19

An answer for when $X$ is a discrete random variable fo finite support and $Y$ is a continuous random variable:

One has to be careful with the definitions as (unlike in the discrete case) there is no such thing as the "entropy" of continuous random variables, and one can only talk about the differential entropy of their densities with respect to certain reference measures (usually the Lebesgue ones). Anyway, with any reasonable definition the difference $H(Y,Z)-H(Y|Z)$ should coincide with the entropy of the condition $H(Z)$. By the basic properties of the entropy $$ H(Z) \le H(X_1) + H(X_2) \;, $$ and the equality holds if and only if the map $$ (x_1,x_2)\mapsto c_1 x_1 + c_2 x_2 $$ is one-to-one on a set of full measure. Since the distribution of $X_i$ has a finite support, one can always choose the constants $c_1,c_2$ in such a way that the latter condition is satisfied. The same argument works for an arbitrary finite number of summands as well.

  • $\begingroup$ the answer is correct if $X$ has a finite support, which is not necessarily the case. Moreover, the definition of $H(X)$ is problematic, as you mentioned, due to the continuous nature of $X$. $\endgroup$ – Jeff May 2 '18 at 9:01
  • 2
    $\begingroup$ If you have a different question then ask it separately. Concerning "finite support" which is "not necessarily the case now" - you had explicitly imposed this condition in the original version. Once again - manipulations like this are not a good practice. $\endgroup$ – R W May 2 '18 at 9:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.