maximum likelihood estimation of X is better than that of f(X)?

Question

Consider a binary variable $C$ with $\Pr(C=0)=\Pr(C=1)=0.5$. Consider a random vector $X \in \mathbb{R}^d$, characterized by $C$, with PDF $p_m(x)$, $m\in\{0,1\}$. Define a maximum likelihood (ML) detector to detect $C$ from $X$. Consider a deterministic mapping $f: \mathbb{R}^d \mapsto \mathbb{R}^k$ for $k > d$, and define $Y = f(X)$. Now define a new ML based on $Y$ to detect $C$. So,

\begin{align} &\text{ML1}: ~C \rightarrow X \rightarrow \hat{C} \\ &\text{ML2}: ~C \rightarrow X \rightarrow Y \rightarrow \hat{C} \end{align}

Is it possible to have a lower detection error probability, namely $\Pr(C\neq \hat{C})$, in the second case? Seems not.

• Assume that all the required probabilities are available, no learning setup.

Answer 1

I interpret the question as follows:

Given are two pdf's, $p_0$ and $p_1$, on $\mathbb{R}^d$ and a function $f\colon \mathbb{R}^d \to \mathbb{R}^k$. For any $x\in\mathbb{R}^d$, let
\begin{equation} \hat c(x)\;\begin{cases} =1&\text{ if } p_1(x)>p_0(x),\\ =0&\text{ if } p_1(x)<p_0(x),\\ \in\{0,1\}&\text{ if } p_1(x)=p_0(x), \end{cases} \end{equation} so that $\hat c(x)$ is the value -- given an observation $x$ -- of a maximum likelihood estimator (MLE) of the unknown parameter $c\in\{0,1\}$ that determines the pdf $p_c$. Suppose next that for each $c\in\{0,1\}$ there exists a pdf $q_c$ of the random variable (r.v.) $Y=f(X)$ assuming that the pdf of the r.v. $X$ is $p_c$. For any $x\in\mathbb{R}^d$, let
\begin{equation} \tilde c(y)\;\begin{cases} =1&\text{ if } q_1(y)>q_0(y),\\ =0&\text{ if } q_1(y)<q_0(y),\\ \in\{0,1\}&\text{ if } q_1(y)=q_0(y), \end{cases} \end{equation} so that $\tilde c(y)$ is the value -- given an observation $y=f(x)$ -- of an MLE of the unknown parameter $c\in\{0,1\}$. Consider the Bayes risks \begin{align*}R(\hat c)&:=\tfrac12\,P_0(\hat c(X)=1)+\tfrac12\,P_1(\hat c(X)=0) \\ &=\tfrac12\,E_0\hat c(X)+\tfrac12-\tfrac12\,E_1\hat c(X) \end{align*} vs. \begin{align*} R(\tilde c)&:=\tfrac12\,P_0(\tilde c(f(X))=1)+\tfrac12\,P_1(\tilde c(f(X))=0) \\ &=\tfrac12\,E_0\tilde c(f(X))+\tfrac12-\tfrac12\,E_1\tilde c(f(X)), \end{align*} where $P_c$ and $E_c$ are respectively the probability and expectation computed assuming that $p_c$ is the pdf of $X$.

The question then appears to be the following: Can $R(\hat c)$ be greater than $R(\tilde c)$?

The answer is now clearly No, since $\hat c$ is a Neyman--Pearson test, and hence most powerful. Here is a brief proof: \begin{align*} 2R(\tilde c)-2R(\hat c) &= E_1\hat c(X)-E_1\tilde c(f(X))-E_0\hat c(X)+E_0\tilde c(f(X)) \\ & = \int[\hat c(x)p_1(x)-\tilde c(f(x))p_1(x)-\hat c(x)p_0(x)+\tilde c(f(x))p_0(x)]\,dx \\ &=\int[(\hat c(x)-\tilde c(f(x)))(p_1(x)-p_0(x))]\,dx\ge0, \end{align*} since $(\hat c(x)-\tilde c(f(x)))(p_1(x)-p_0(x))\ge0$ for all $x$.