Mutual information between continuous and discrete variables from numerical data

Question

I am looking for references/measures to estimate the mutual information between a continuous (C) and discrete (D) variable, given a real-world (i.e. finite sample) data set. C is uniformly distributed between Cmin and Cmax, and D is a positive integer whose distribution depends on the value of C (e.g., a poisson distribution with mean f(C)). Any suggestions (methods based on binning over the data set, or independent of binning) are welcome.

Answer 1

$\newcommand{\de}{\delta} \newcommand{\ep}{\varepsilon} \newcommand{\thh}{\theta}$ Let $D$ be any discrete random variable (r.v.) with distinct values $d_i$ taken with probabilities $p_i=P(D=d_i)>0$ for $i\in I$, where $I$ is a denumerable (that is, at most countable) set. Let $X$ be any r.v. (defined on the same probability space as $D$), with values in any nonempty set $S$ (given also some sigma-algebra $\Sigma$ over $S$, to make $S$ a measurable space). Let $\mu$ be the probability distribution of $X$, so that $\mu(B)=P(X\in B)$ for all $B\in\Sigma$. For each $i\in I$ and each $B\in\Sigma$, let \begin{equation*} \mu_i(B):=P(D=d_i,X\in B). \end{equation*} Then $\mu_i$ is a (sub-probability) measure absolutely continuous with respect to $\mu$, so that we can consider a Radon--Nikodym density \begin{equation*} \rho_i:=\frac{d\mu_i}{d\mu} \tag{1} \end{equation*} of the measure $\mu_i$ with respect to $\mu$, so that the values of $\rho_i$ are in $[0,1]$.

Then the mutual information between $D$ and $X$ may/should be defined as follows: \begin{equation*} I(D,X):=\sum_{i\in I}\int_S d\mu\;\rho_i\ln\frac{\rho_i}{p_i}. \tag{!} \end{equation*}

Justification: Take any natural $n$. Let $J:=J_n:=\{0,\dots,n\}^I$, the set of families $j=(j_i)_{i\in I}$ with $j_i\in\{0,\dots,n\}$. Let us assume that $I$ is finite, so that $J$ is countable. For any $j\in J$, let \begin{equation*} \de_j:=\de_j(n):=\{x\in S\colon j_i/n\le\rho_i(x)<(j_i+1)/n\ \forall i\in I\}. \tag{2} \end{equation*} Let $J_{n;0}:=\{j\in J\colon\de_j\ne\emptyset\}$. Note that the sets $\de_j$ with $j\in J_{n;0}$ form a partition of $S$. For each $j\in J_{n;0}$, pick then an arbitrary $x_j\in\de_j$, and let $X_n$ be the r.v. taking value the constant value $x_j$ on $\de_j$ for each $j\in J_{n;0}$, so that $X_n$ is a discrete approximation to the r.v. $X$. Then, according to the standard definition, the mutual information between $D$ and $X_n$ is \begin{align} I(D,X_n)&:=\sum_i\sum_j P(D=d_i,X_n=x_j)\ln\frac{P(D=d_i,X_n=x_j)}{P(D=d_i)P(X_n=x_j)} \\ &=\sum_i\sum_j P(D=d_i,X\in\de_j)\ln\frac{P(D=d_i,X\in\de_j)}{P(D=d_i)P(X\in\de_j)} \\ &=\sum_i\sum_j \mu_i(\de_j)\ln\frac{\mu_i(\de_j)}{p_i\mu(\de_j)}; \end{align} here and in what follows, $\sum_i:=\sum_{i\in I}$ and $\sum_j:=\sum_{j\in J_{n;0}}$; as usual, the $(i,j)$-summand here is understood as $0$ whenever $\mu_i(\de_j)=0$. Next, by (1) and (2), \begin{equation*} \frac{\mu_i(\de_j)}{\mu(\de_j)}=\frac1{\mu(\de_j)}\,\int_{\de_j}d\mu\;\rho_i =\rho_i(x_j)+\thh_{ij}/n \end{equation*} for some $\thh_{ij}\in[0,1]$. So, \begin{align} I(D,X_n)&=\sum_i\sum_j [\rho_i(x_j)+\thh_{ij}/n]\mu(\de_j)\ln\frac{\rho_i(x_j)+\thh_{ij}/n}{p_i} \\ &=\sum_i\sum_j \rho_i(x_j)\mu(\de_j)\ln\frac{\rho_i(x_j)}{p_i}+O(1/n) \\ &\to\sum_i\int_S d\mu\;\rho_i\ln\frac{\rho_i}{p_i} \end{align} as $n\to\infty$, which completes the justification of the definition (!).