I asked this question on stats.stackexchange and even elsewhere, but it never received an answer.

I just state the probabilistic problem here. It is about the optimality of the conditional expectation.

Consider a random vector $(X_1,X_2,X_3,Y)$ with $Y\in L^1$, and let $f$ be the Borelian function such that $E[Y\mid X_1, X_2, X_3]=f(X_1, X_2, X_3)$.

Define $$I = E\left[{\bigl(Y-f(X'_1,X_2,X_3)\bigr)}^2\right] - E\left[{\bigl(Y-f(X_1,X_2,X_3)\bigr)}^2\right]$$ where $X'_1$ is a random variable having the same distribution as $X_1$ but is independent of all other random variables $X_2,X_3,Y$.

The question: "is it possible that $I < 0$?"

In case you are interested in a simple example, the paper Correlation and variable importance in random forests (Gregorutti & al) investigates some particular cases such as the additive case $f(X_1,X_2,X_3)=f_1(X_1)+f_2(X_2)+f_3(X_3)$. It is not difficult to get that $I=2\textrm{Var}\bigl(f_1(X_1)\bigr)$ in this case.


That is not possible: $I$ cannot be negative.

Indeed, let $X:=X_1$, $X':=X'_1$, $V:=(X_2,X_3)$. Let $E_Z$ denote the conditional expectation given a random variable/random vector $Z$. Then $E_{(X',V)}Y=E_V Y$, since $X'$ is independent of $(Y,V)$ (see details below). So, $$E(Y-f(X',V))^2\ge E(Y-E_{(X',V)}Y)^2=E(Y-E_V Y)^2 \ge E(Y-E_{(X,V)} Y)^2=E(Y-f(X,V))^2. $$ The two inequalities above hold because $E(Y-g(Z))^2\ge E(Y-E_Z Y)^2$ for any Borel function $g$.

Details on the equality $E_{(X',V)}Y=E_V Y$: Let $h(V):=E_V Y$. Let $I\{\cdot\}$ denote the indicator function. Let $A$ and $B$ denote Borel sets in appropriate spaces. Then, by the independence of $X'$ from $(Y,V)$, $$EYI\{X'\in A\}I\{V\in B\}=EI\{X'\in A\}\,EYI\{V\in B\}$$ $$=EI\{X'\in A\}\,Eh(V)I\{V\in B\} =Eh(V)I\{X'\in A\}I\{V\in B\}, $$ whence $E_{(X',V)}Y=h(V)=E_V Y$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.