Justification of the use of residual plot

Question

$\DeclareMathOperator\Cov{Cov}$Backround of my Question

Let $Y$ be the response variable, $\mathbb{X}$ be the explanatory variables. The ultimate goal of prediction is finding a function $f^{*}$ that minimize $\mathbb{E}[(Y - f^{*}(\mathbb{X})^2)]$, we know that the solution is $f^{*}(\mathbb{X}) = \mathbb{E}[Y | \mathbb{X}]$. Let $\epsilon = Y - \mathbb{E}[Y | \mathbb{X}]$, then we have $$ Y = f^{*}(\mathbb{X}) + \epsilon $$ where $\mathbb{E}[\mathbb{\epsilon}] = 0$, and $\Cov[\mathbb{X}, \epsilon] = 0$.

So, I think the goal of regression somehow become finding a function $g \approx f^*$ from some hypothesis space (e.g. $g(\mathbb{X}) = \mathbb{X}\beta$ for linear regression).

One way of defining $g \approx f^*$ (which can persuade myself that it is a good approximation) is $$ \mathbb{P}[|f^{*}(\mathbb{X}) - g(\mathbb{X})| > \eta] < \delta $$ for small $\eta$ and $\delta$.

Question

Given that $g \approx f^{*}$ (in the sense defined above), if I can show that $\Cov[\widehat{Y}, \widehat{\epsilon}] \approx 0$ ($\widehat{Y} = g(\mathbb{X})$, and $\widehat{\epsilon} = Y - g(\mathbb{X})$), then when I see a residual plot like this Residual Plot

I can persuade myself this might be a signal suggesting that $g(\mathbb{X})$ probably a good approximation.

So, my question is how to show that $g \approx f^* \Longrightarrow \Cov[\widehat{Y}, \widehat{\epsilon}] \approx 0$?

My Attemp

If further assume that $\mathbb{X}$ and $\epsilon$ are independent (is $\Cov[\mathbb{X}, \epsilon] = 0$ sufficient?), then $\Cov[f^{*}\mathbb(X), \epsilon] = 0$, hence

\begin{equation} \begin{aligned} |\Cov[\widehat{Y}, \widehat{\epsilon}]| &= |\Cov[f^*(\mathbb{X}) + (g(\mathbb{X}) - f^*\mathbb(X)), (f^*(\mathbb{X}) - g(\mathbb{X})) + \epsilon]| \\ &\leq |\Cov[f^{*}(\mathbb{X}), f^{*}(\mathbb{X}) - g(\mathbb{X})]| + |\Cov[g(\mathbb{X}) - f^{*}(\mathbb{X}), f^{*}(\mathbb{X}) - g(\mathbb{X})]| + |\Cov[g(\mathbb{X}) - f^{*}(\mathbb{X}), \epsilon]| \\ & \leq |\Cov[f^{*}(\mathbb{X}), f^{*}(\mathbb{X}) - g(\mathbb{X})]| + |\Cov[g(\mathbb{X}) - f^{*}(\mathbb{X}), \epsilon]| + |Var[f^*(\mathbb{X}) - g(\mathbb{X})]| \end{aligned} \end{equation} Intuitively, since $f^{*}(\mathbb{X}) - g(\mathbb{X}) \approx 0$ with high probability, the last three terms may be small (depend on $\epsilon$ and $\delta$). But I don't know how to do it formally.

Edit

By @losifpinelis 's construction, $\exists (Y, \mathbb{X})$, for any given $\eta, \delta, M > 0$, there exists a function $g_N$ such that $$ \mathbb{P}[|f^{*}(\mathbb{X}) - g_N(\mathbb{X})| > \eta] < \delta $$ but $|\Cov[g_N(\mathbb{X}), \epsilon_N]| > M$. Therefore, this is not a proper definition for this problem.

My next Question is "Is $\mathbb{E}[(f^{*}(\mathbb{X}) - g(\mathbb{X}))^2] < \eta$ a proper definition of goodness of approximation?". That is, does $$ \mathbb{E}[(f^{*}(\mathbb{X}) - g(\mathbb{X}))^2] < \eta \Longrightarrow \Cov[\widehat{Y}, \widehat{\epsilon}] < SomeFunction(\eta) $$ hold?

Answer 1

$\newcommand\ep\epsilon\newcommand{\de}{\delta}$Getting rid of the instances of $\approx$, one can state the question as follows:

Let $f(X):=E(Y|X)$, where $X$ and $Y$ are random variables (r.v.'s) ($X$ possibly a multivariate one) such that $EY^2<\infty$. Note that $E\ep=0$ and $Cov(f(X),\ep)=0$, where $\ep:=Y-f(X)$.

Suppose that for a sequence $(g_n)$ of Borel-measurable functions one has $g_n(X)\to f(X)$ in probability (as $n\to\infty$). Does it then follow that \begin{equation*} Cov(g_n(X),\ep_n)\to0, \tag{1}\label{1} \end{equation*} where $\ep_n:=Y-g_n(X)$?

The answer to this question is: Of course, not.

Indeed, let e.g. $X$ be a r.v. uniformly distributed on the interval $[-1,1]$, and let $Y:=X$, so that $f(X)=X$ and $\ep=0$.

Let \begin{equation*} g_n(X):=X\,1(|X|\ge1/n)+n^2 X\,1(|X|<1/n). \end{equation*} Then $g_n(X)\to X=f(X)$ in probability.

However, $Eg_n(X)=0$, $\ep_n=Y-g_n(X)=X-g_n(X)=(1-n^2)X\,1(|X|<1/n)$, and hence \begin{equation*} Cov(g_n(X),\ep_n)=Eg_n(X)\ep_n=n^2(1-n^2)\,EX^2\,1(|X|<1/n) \\ =(1-n^2)/(3n)\to-\infty\ne0. \end{equation*} So, \eqref{1} fails to hold. $\quad\Box$

---

On the other hand, if $g_n(X)\to f(X)$ in $L^2$, then it is easy to see that \eqref{1} will hold.

Details on this: We have $\|g_n(X)-f(X)\|_2\to0$, where $\|Z\|_2:=\sqrt{EZ^2}$. We also have the identity \begin{equation*} g_n(X)\ep_n=g_n(X)(Y-g_n(X))=f(X)(Y-f(X))+Y(g_n(X)-f(X))+2f(X)(f(X)-g_n(X))-(g_n(X)-f(X))^2. \end{equation*} Taking here the expectations and recalling that $Ef(X)(Y-f(X))=Ef(X)\ep=Cov(f(X),\ep)=0$, we get \begin{equation*} Eg_n(X)\ep_n=EY(g_n(X)-f(X))+2Ef(X)(f(X)-g_n(X))-E(g_n(X)-f(X))^2. \tag{2}\label{2} \end{equation*} By the Cauchy--Schwarz inequality and the condition $\|g_n(X)-f(X)\|_2\to0$, \begin{equation*} |EY(g_n(X)-f(X))|\le\|Y\|_2\,\|g_n(X)-f(X)\|_2\to0. \tag{3}\label{3} \end{equation*} Note all that \begin{equation} \|f(X)\|_2\le\|Y\|_2<\infty \tag{3.5}\label{3.5} \end{equation} since $f(X)$ is an orthogonal projection of $Y$ in $L^2$. So, \begin{equation*} |Ef(X)(f(X)-g_n(X))|\le\|f(X)\|_2\,\|g_n(X)-f(X)\|_2\to0. \tag{4}\label{4} \end{equation*} Also, \begin{equation*} E(g_n(X)-f(X))^2=\|g_n(X)-f(X)\|_2^2\to0. \tag{5}\label{5} \end{equation*} Collecting \eqref{2}--\eqref{5}, we get \begin{equation*} Eg_n(X)\ep_n\to0. \tag{6}\label{6} \end{equation*} Next, $\ep_n=Y-g_n(X)=Y-f(X)+f(X)-g_n(X)=\ep+f(X)-g_n(X)$. Taking here the expectations and recalling that $E\ep=0$, we get \begin{equation} |E\ep_n|=|E(f(X)-g_n(X))|\le\|g_n(X)-f(X)\|_2\to0. \tag{7}\label{7} \end{equation} Further, $|Eg_n(X)-Ef(X)|\le E|g_n(X)-Ef(X)|\le\|g_n(X)-Ef(X)\|_2\to0$, again by the Cauchy--Schwarz inequality and the condition $\|g_n(X)-f(X)\|_2\to0$. So, \begin{equation} Eg_n(X)\to Ef(X), \tag{8}\label{8} \end{equation} and $|Ef(X)|\le\|f(X)\|_2<\infty$ by the Cauchy--Schwarz inequality and \eqref{3.5}. By \eqref{6}--\eqref{8}, \begin{equation} Cov(g_n(X),\ep_n)=Eg_n(X)\ep_n-Eg_n(X)\,E\ep_n\to0-Ef(X)\times 0=0, \end{equation}

---

In view of \eqref{3}, \eqref{3.5}, \eqref{4}, \eqref{5}, and \eqref{7}, one can also get an explicit bound on $|Cov(g_n(X),\ep_n)|$: \begin{equation} |Cov(g_n(X),\ep_n)|\le3y\de_n+2\de_n^2, \end{equation} where $y:=\|Y\|_2$ and $\de_n:=\|g_n(X)-f(X)\|_2$.