2
$\begingroup$

I am looking for references/measures to estimate the mutual information between a continuous (C) and discrete (D) variable, given a real-world (i.e. finite sample) data set. C is uniformly distributed between Cmin and Cmax, and D is a positive integer whose distribution depends on the value of C (e.g., a poisson distribution with mean f(C)). Any suggestions (methods based on binning over the data set, or independent of binning) are welcome.

$\endgroup$
  • $\begingroup$ Since you already know the prior distribution of C, it would be probably easier to estimate the conditional entropy of C given D to estimate mutual information. (Curiously, this is something you may encounter in neuroscience applications for spike counts.) As for the binning question, MO is probably not the best place to ask since it is not really a math question. Try stats.SE? $\endgroup$ – Memming Jan 21 at 13:24
  • $\begingroup$ Thanks for the clue. Will try implementing it. $\endgroup$ – Gaurang Jan 23 at 7:45
1
$\begingroup$

$\newcommand{\de}{\delta} \newcommand{\ep}{\varepsilon} \newcommand{\thh}{\theta}$ Let $D$ be any discrete random variable (r.v.) with distinct values $d_i$ taken with probabilities $p_i=P(D=d_i)>0$ for $i\in I$, where $I$ is a denumerable (that is, at most countable) set. Let $X$ be any r.v. (defined on the same probability space as $D$), with values in any nonempty set $S$ (given also some sigma-algebra $\Sigma$ over $S$, to make $S$ a measurable space). Let $\mu$ be the probability distribution of $X$, so that $\mu(B)=P(X\in B)$ for all $B\in\Sigma$. For each $i\in I$ and each $B\in\Sigma$, let \begin{equation*} \mu_i(B):=P(D=d_i,X\in B). \end{equation*} Then $\mu_i$ is a (sub-probability) measure absolutely continuous with respect to $\mu$, so that we can consider a Radon--Nikodym density \begin{equation*} \rho_i:=\frac{d\mu_i}{d\mu} \tag{1} \end{equation*} of the measure $\mu_i$ with respect to $\mu$, so that the values of $\rho_i$ are in $[0,1]$.

Then the mutual information between $D$ and $X$ may/should be defined as follows: \begin{equation*} I(D,X):=\sum_{i\in I}\int_S d\mu\;\rho_i\ln\frac{\rho_i}{p_i}. \tag{!} \end{equation*}

Justification: Take any natural $n$. Let $J:=J_n:=\{0,\dots,n\}^I$, the set of families $j=(j_i)_{i\in I}$ with $j_i\in\{0,\dots,n\}$. Let us assume that $I$ is finite, so that $J$ is countable. For any $j\in J$, let \begin{equation*} \de_j:=\de_j(n):=\{x\in S\colon j_i/n\le\rho_i(x)<(j_i+1)/n\ \forall i\in I\}. \tag{2} \end{equation*} Let $J_{n;0}:=\{j\in J\colon\de_j\ne\emptyset\}$. Note that the sets $\de_j$ with $j\in J_{n;0}$ form a partition of $S$. For each $j\in J_{n;0}$, pick then an arbitrary $x_j\in\de_j$, and let $X_n$ be the r.v. taking value the constant value $x_j$ on $\de_j$ for each $j\in J_{n;0}$, so that $X_n$ is a discrete approximation to the r.v. $X$. Then, according to the standard definition, the mutual information between $D$ and $X_n$ is \begin{align} I(D,X_n)&:=\sum_i\sum_j P(D=d_i,X_n=x_j)\ln\frac{P(D=d_i,X_n=x_j)}{P(D=d_i)P(X_n=x_j)} \\ &=\sum_i\sum_j P(D=d_i,X\in\de_j)\ln\frac{P(D=d_i,X\in\de_j)}{P(D=d_i)P(X\in\de_j)} \\ &=\sum_i\sum_j \mu_i(\de_j)\ln\frac{\mu_i(\de_j)}{p_i\mu(\de_j)}; \end{align} here and in what follows, $\sum_i:=\sum_{i\in I}$ and $\sum_j:=\sum_{j\in J_{n;0}}$; as usual, the $(i,j)$-summand here is understood as $0$ whenever $\mu_i(\de_j)=0$. Next, by (1) and (2), \begin{equation*} \frac{\mu_i(\de_j)}{\mu(\de_j)}=\frac1{\mu(\de_j)}\,\int_{\de_j}d\mu\;\rho_i =\rho_i(x_j)+\thh_{ij}/n \end{equation*} for some $\thh_{ij}\in[0,1]$. So, \begin{align} I(D,X_n)&=\sum_i\sum_j [\rho_i(x_j)+\thh_{ij}/n]\mu(\de_j)\ln\frac{\rho_i(x_j)+\thh_{ij}/n}{p_i} \\ &=\sum_i\sum_j \rho_i(x_j)\mu(\de_j)\ln\frac{\rho_i(x_j)}{p_i}+O(1/n) \\ &\to\sum_i\int_S d\mu\;\rho_i\ln\frac{\rho_i}{p_i} \end{align} as $n\to\infty$, which completes the justification of the definition (!).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.