Orthogonal decomposition of conditional expectations

Question

Suppose I have a random variable $x$ and a set of conditional distributions on $x$. Here is an example where the conditionals are nested: $$q_1 := E(x|y_1), \quad q_2 := E(x|y_1,y_2),\quad q_3 := E(x|y_1,y_2,y_3)$$ What I'd like to find is an orthogonal decomposition of $x$. For the example above, we have: $$ x = q_1 \oplus (q_2-q_1) \oplus (q_3-q_2) \oplus (x-q_3) $$ Is there a way to perform such an orthogonal decomposition when the conditionals are not nested? For example, what if we had: $$q_1 := E(x|y_1),\quad q_2 := E(x|y_1,y_2),\quad q_3 := E(x|y_1,y_3)$$

Answer 1

What about the standard Gram-Schmidt procedure?

Given any collection of expectations $q_1, q_2, \ldots, q_n$, we can write: $$ X = q_1 + (q_2-q_1) + \cdots + (q_n-q_{n-1}) + (X-q_n) $$ Now define $v_1=q_1$, $v_2 = q_2-q_1$, ..., $v_n=q_n-q_{n-1}$, $v_{n+1} = X-q_n$, so $X = \sum_{i=1}^{n+1} v_i$. The inner product between $v_i$ and $v_j$ is $E[v_iv_j]$, and $||v_i||^2 = E[v_i^2]$. So by Gram-Schmidt we can define: \begin{align} u_1 &= \frac{v_1}{||v_1||} \\ u_2 &= \frac{v_2 - E[v_2u_1]u_1}{||numerator||} \\ u_3 &= \frac{v_3 - E[v_3u_1]u_1 - E[v_3u_2]u_2}{||numerator||} \end{align} and so on, assuming at each step that the denominator is nonzero (else, we throw that particular vector away as it is a linear combination of the others). Then $E[u_iu_j]=0$ for all $i \neq j$, and $X = \sum_{i=1}^{n+1} E[Xu_i]u_i$.

---

For your example: $q_1=E[X|Y_1], q_2 = E[X|Y_1,Y_2], q_3 = E[X|Y_1,Y_3]$, so: \begin{align} v_1 &= E[X|Y_1]\\ v_2 &= E[X|Y_1,Y_2] - E[X|Y_1]\\ v_3 &= E[X|Y_1,Y_3] - E[X|Y_1,Y_2]\\ v_4 &= X - E[X|Y_1,Y_3] \end{align} and: \begin{align} u_1 &= \frac{E[X|Y_1]}{\sqrt{E[v_1^2]}} \\ u_2 &= \frac{v_2 - E[v_2u_1]u_1}{||numerator||} = \frac{v_2}{\sqrt{E[v_2^2]}}\\ u_3 &= \frac{v_3 - E[v_3u_1]u_1 - E[v_3u_2]u_2}{||numerator||} = \frac{v_3-E[v_3u_2]u_2}{||numerator||}\\ u_4 &= \frac{v_4 - E[v_4u_1]u_1 - E[v_4u_2]u_2-E[v_4u_3]u_3}{||numerator||} = \frac{v_4 - E[v_4u_2]u_2 - E[v_4u_3]u_3}{||numerator||} \end{align} and so $\{u_1, \ldots, u_4\}$ are orthonormal and $X = \sum_{i=1}^4 E[Xu_i]u_i$.