Consider continuous random variable $X$ with bounded support $\mathcal{X} \subset \mathbb{R}$, discrete random variable $Y$ with finite support $\mathcal{Y}$ , and constants $c_1 ,c_2 \in \mathbb{R}$. Let $p_{XY}(x,y)$ be the probability measure on $\mathcal{X} \times \mathcal{Y}$. Define $H(X,Y)$ and $H(Y\mid X)$ as the joint and conditional entropies. Define $Z = c_1 X_1 + c_2 X_2$ where $X_1$ and $X_2$ are i.i.d. realizations from the probability measure on $X$. What are $c_1,c_2$ that maximize $I(Y;Z)$ where $I$ is the mutual information? what about general case with $n$ constants $c_1,...,c_n$?
$\begingroup$
$\endgroup$
7
-
$\begingroup$ You mean $Z$ instead of $RX$, don't you? $\endgroup$– Jean DuchonMay 1, 2018 at 10:10
-
$\begingroup$ yes, I have corrected the question. $\endgroup$– JeffMay 1, 2018 at 10:58
-
$\begingroup$ Trivial if you can make the map invertible? Can I choose a $c_1$ and $c_2$ such that you can decode back $X_1$ and $X_2$ from $Z$? $\endgroup$– MemmingMay 1, 2018 at 13:07
-
$\begingroup$ @Memming the map might not be invertible in general. $\endgroup$– JeffMay 2, 2018 at 7:18
-
$\begingroup$ @MichaelHardy, you are right and that was a typo. I have corrected the question. $\endgroup$– JeffMay 2, 2018 at 7:19
|
Show 2 more comments
1 Answer
$\begingroup$
$\endgroup$
2
One has to be careful with the definitions as (unlike in the discrete case) there is no such thing as the "entropy" of continuous random variables, and one can only talk about the differential entropy of their densities with respect to certain reference measures (usually the Lebesgue ones). Anyway, with any reasonable definition the difference $H(Y,Z)-H(Y|Z)$ should coincide with the entropy of the condition $H(Z)$. By the basic properties of the entropy $$ H(Z) \le H(X_1) + H(X_2) \;, $$ and the equality holds if and only if the map $$ (x_1,x_2)\mapsto c_1 x_1 + c_2 x_2 $$ is one-to-one on a set of full measure. Since the distribution of $X_i$ has a finite support, one can always choose the constants $c_1,c_2$ in such a way that the latter condition is satisfied. The same argument works for an arbitrary finite number of summands as well.
-
$\begingroup$ the answer is correct if $X$ has a finite support, which is not necessarily the case. Moreover, the definition of $H(X)$ is problematic, as you mentioned, due to the continuous nature of $X$. $\endgroup$– JeffMay 2, 2018 at 9:01
-
2$\begingroup$ If you have a different question then ask it separately. Concerning "finite support" which is "not necessarily the case now" - you had explicitly imposed this condition in the original version. Once again - manipulations like this are not a good practice. $\endgroup$– R WMay 2, 2018 at 9:05