$E(X_1 | X_1 + X_2)$, where $X_i$ are (integrable) independent infinitely divisible rv's “of the same type”

The following is inspired by this recent question on math.stackexchange. Two standard exercises in conditional expectation are to find ${\rm E}(X_1|X_1+X_2)$ where: 1) $X_i$, $i=1,2$, are independent ${\rm N}(0,\sigma_i^2)$ rv's; 2) $X_i$, $i=1,2$, are independent Poisson($\lambda_i$) rv's. The solutions are given by $\frac{{\sigma _1^2 }}{{\sigma _1^2 + \sigma _2^2 }}(X_1 + X_2)$ and $\frac{{\lambda _1 }}{{\lambda _1 + \lambda _2 }}(X_1 + X_2)$, respectively. A proof for case 1) is given on math.stackexchange. For case 2) we have ${\rm E}(X_1|X_1 + X_2 = n) = \sum\limits_{k = 0}^n {k{\rm P}(X_1 = k|X_1 + X_2 = n)}.$ A straightforward calculation shows that the right-hand side sum is equal to $\sum\limits_{k = 0}^n {k{n \choose k}\bigg(\frac{{\lambda _1 }}{{\lambda _1 + \lambda _2 }}\bigg)^k \bigg(\frac{{\lambda _2 }}{{\lambda _1 + \lambda _2 }}\bigg)^{n - k} },$ which is the expectation of the binomial distribution with parameters $n$ and $\lambda_1 / (\lambda_1 + \lambda_2)$, hence given by $n \lambda_1 / (\lambda_1 + \lambda_2)$. The result for case 2) is thus proved. In this context, what is common to the normal and Poisson distributions is that both are infinitely divisible (ID). More specifically, the characteristic function of $X_i \sim {\rm N}(0,\sigma_i^2)$ is given by ${\rm E}[{\rm e}^{{\rm i}zX_i} ] = {\rm e}^{\sigma _i^2 ( - z^2 /2)}$, and that of $X_i \sim {\rm Poisson}(\lambda_i)$ by ${\rm E}[{\rm e}^{{\rm i}zX_i} ] = {\rm e}^{\lambda _i ({\rm e}^{{\rm i}z} - 1)}$. Now, consider integrable, ID, independent rv's $X_i$, $i=1,2$, with characteristic functions of the form ${\rm E}[{\rm e}^{{\rm i}zX_i} ] = {\rm e}^{c_i \psi(z)}$, $c_i > 0$ (loosely speaking, the characteristic function of an arbitrary ID rv is of that form). In view of the normal and Poisson examples considered above (the former requires somewhat tedious algebra for the solution), and the fact that many important rv's fall into the general category of integrable ID rv's (e.g., gamma rv's), it would be very useful to have the following result: ${\rm E}(X_1 | X_1 + X_2) = \frac{{c _1 }}{{c_1 + c_2 }}(X_1 + X_2)$. In fact, I have proved it recently. Now to my questions: 1) Have you encountered this result before? 2) Can you provide a rigorous but simple proof of it? 3) Can you provide some intuition?

EDIT: 1) Here's another interesting example: if $X_i \sim {\rm Gamma}(c_i,\lambda)$, $i=1,2$, so that $X_i$ has density $f_{X_i } (x) = \lambda ^{c_i } {\rm e}^{ - \lambda x} x^{c_i - 1} /\Gamma (c_i )$, $x > 0$, then ${\rm E}(X_1 | X_1 + X_2) = \frac{{c_1 }}{{c_1 + c_2 }}(X_1 + X_2 )$. 2) It is very instructive to reformulate the result in terms of L\'evy processes: if $X = \{ X(t): t \geq 0 \}$ is an integrable L\'evy process, then ${\rm E}[X(s)|X(t)] = \frac{s}{t}X(t)$, $0 < s < t$.

EDIT: The "direct" solution for the gamma case considered above is now given here. This shows, once more, the effectiveness of the general formula.

EDIT: A complete solution is given in my first (according to date) answer below.

EDIT: An important extension is considered in my second answer below.

-
In the two concrete examples you cite, the $c_i$s are the variances. Do you have other interesting concrete cases where that happens and where it doesn't? –  Michael Hardy Nov 24 '10 at 1:09
Your observation concerning the variances seems to be related to the following facts. If $\mu$ is an infinitely divisible distribution, then there exists a unique L\'evy process $X = \{X_t:t \geq 0\}$ such that $\mu$ is the distribution of $X_1$. If $X_1$ is square-integrable, then $X_t$ is square-integrable for any $t > 0$, and in fact ${\rm Var}(X_t)$ is linear in $t$ ($= \sigma^2 t$ for Brownian motion with variance parameter $\sigma^2$, $= \lambda t$ for Poisson process with parameter $\lambda$). ${\rm E}(X_t)$ is also linear in $t$, but in the centered normal case it is identically $0$ –  Shai Covo Nov 24 '10 at 2:17

Regarding a "rigorous but simple proof" of the relation the OP is interested in, such a proof is, almost completely, already written in the original post.

To see this, consider independent integrable random variables $X$ and $Y$ and assume that their characteristic functions, defined for every real number $t$, are such that $E({\mathrm e}^{\mathrm{i}tX})=\mathrm{e}^{a\psi(t)}$ and $E(\mathrm{e}^{\mathrm{i}tY})=\mathrm{e}^{b\psi(t)}$ for a given function $\psi$ and given real numbers $a$ and $b$. Let $S=X+Y$. Now, to prove that $$(a+b)E(X\vert S)=aS,$$ it suffices to show that, for every real number $t$, $$(a+b)E(X\mathrm{e}^{\mathrm{i}tS})=aE(S\mathrm{e}^{\mathrm{i}tS}).$$ Since both sides of the equality can be explicitly written in terms of $a$, $b$, the function $\psi$ and its derivative $\psi'$, the proof is, in a way and modulo some easy computations, already over.

For example, $$E(S\mathrm{e}^{\mathrm{i}tS})=E(X\mathrm{e}^{\mathrm{i}tX})E({\mathrm e}^{\mathrm{i}tY})+E(Y\mathrm{e}^{\mathrm{i}tY})E({\mathrm e}^{\mathrm{i}tX}),$$ because $X$ and $Y$ are independent. Here, both $E({\mathrm e}^{\mathrm{i}tX})$ and $E({\mathrm e}^{\mathrm{i}tY})$ are already known, and both $E(X{\mathrm e}^{\mathrm{i}tX})$ and $E(Y{\mathrm e}^{\mathrm{i}tY})$ are derivatives of the former with respect to $(\mathrm{i}t)$. Hence, $$E(S\mathrm{e}^{\mathrm{i}tS})=-\mathrm{i}(a+b)\psi'(t)\mathrm{e}^{(a+b)\psi(t)}.$$ Likewise, $$E(X\mathrm{e}^{\mathrm{i}tS})=E(X\mathrm{e}^{\mathrm{i}tX})E({\mathrm e}^{\mathrm{i}tY})=-\mathrm{i}a\psi'(t)\mathrm{e}^{(a+b)\psi(t)}.$$ Comparing these two formulas, we are done.

(If this helps, one can note that the signs of $a$ and $b$ must be the same, in the sense that $ab>0$ or that $X$ or $Y$ must be $0$ with full probability.)

-
Thank you. I'll go over your solution today or so. –  Shai Covo Nov 28 '10 at 4:45
Can you please elaborate on "Now, to prove that $(a+b)E(X\vert S)=aS$, it suffices to show that $(a+b)E(X\mathrm{e}^{\mathrm{i}tS})=aE(S\mathrm{e}^{\mathrm{i}tS})$ for every real number $t$". It seems that this is based on a property which is not too well-known; can you provide some reference? –  Shai Covo Dec 1 '10 at 23:49
Your aim is to prove that $(a+b)E(Xg(S))=aE(Sg(S))$ for every bounded measurable $g$, or for every $0,1$-valued measurable $g$, or for every $g$ in a class a functions large enough to recover the preceding ones. Your hypothesis is that this holds for every $g$ defined by $g(x)=\mathrm{e}^{\mathrm{i}tx}$. Hence this holds for every linear combination of these, hence, by density, for every $g$ in the Schwartz space--et voilà. –  Did Dec 2 '10 at 7:27
(cont'd) Or, you may consider the finite signed measures $\mu$ and $\nu$ defined by $\int g\mathrm{d}\mu=(a+b)E(Xg(S))$ and $\int g\mathrm{d}\nu=aE(Sg(S))$ for every bounded measurable $g$ on the real line. You know the Fourier transforms $\hat\mu$ and $\hat\nu$ coincide, hence $\mu=\nu$. (The proof in the preceding comment is a pedestrian rediscovery of this one.) –  Did Dec 2 '10 at 7:27
Thanks for the clarification. –  Shai Covo Dec 6 '10 at 16:49

Suppose $X_1$ and $X_2$ are i.i.d. integrable rv's. Then $\mathbb{E}(X_1 | X_1+X_2)=(X_1 + X_2)/2$ by symmetry. Similarly, if $X_1,\ldots,X_n$ are i.i.d and integrable then $\mathbb{E}(X_1 | \sum X_i)=\sum X_i/n$. Hence if $S=\sum_{i=1}^n X_i$ and $T=\sum_{i=n+1}^{n+m} X_i$ then $$\mathbb{E}(S | S+T)=\frac{n}{n+m}S+T \ .$$

Any two infinitely divisible rv's with rational ratio of parameters can be decomposed like that and the rest follows by continuity.

-
Just a small clarification of the spoken symmetry: $E[X_1|X_1+X_2]=E[X_2|X_1+X_2]$, but $E[X_1|X_1+X_2]+E[X_2|X_1+X_2]=X_1+X_2$, whence the Ori's statement. So this fact is very simple. Though not innocent at all. For example, from $E[X_1|X_1+\dots+X_n]=\frac1n\sum_{i=1}^n X_i$, the Hewitt-Savage 0-1 law and results on convergence of $\sigma$-algebras one gets strong law of large numbers. –  zhoraster Nov 24 '10 at 7:34
(And, clearly, it is true not only for independent but for any exchandeable sequence.) –  zhoraster Nov 24 '10 at 7:36
Thank you Ori and zhoraster. I'll go over the answer/comments later on today. However, I'm not sure how rigorous the proof is. Specifically, "and the rest follows by continuity" needs rigorous justification. –  Shai Covo Nov 24 '10 at 8:56

I think the right way to phrase this discussion is as follows. Let $(X_s)_{0 \leq s \leq t}$ be a real stochastic process with cyclically exchangeable increments: for all $u \in [0,t]$, the process $(X'_s)_{0\leq s \leq t}$ obtained by a cyclic shift by $u$, has the same distribution as the original process.

Suppose that $X_0=X_t=0$ with probability one. Then for all $s$, $\mathbb{E}(X_s)=0$. (As in Ori's argument, for this step a continuity argument is needed, which you may not like. On the other hand, this kind of continuity argument is bog-standard -- it is a basic procedure when you study infinitely divisible distributions via their characteristic functions.)

Edit: Here is an argument to replace the continuity argument but which requires an additional assumption. Suppose for simplicity that $t=1$. The additional assumption is that $\sup_{s \in (0,1)} |\mathbb{E}(X_s)| < \infty$. Suppose there is $s$ s.t. $\mathbb{E}(X_s) = z > 0$. Then by cyclic exchangeability, $\mathbb{E}(X_{1-s}) = -z$. Again by cyclic exchangeability, $|\mathbb{E}(X_{|2s-1|})| = 2z$, the sign depending on the sign of $2s-1$.

By repeating this argument, it follows that if there is any point $s$ with $\mathbb{E}(X_s) \neq 0$ then there are points $s$ for which $|\mathbb{E}(X_s)|$ is arbitrarily large. In fact, since by cyclic exchangeability, $\mathbb{E}(X_{s/n})=\mathbb{E}(X_s)/n$, it then follows that there are points arbitrarily close to zero for which $|\mathbb{E}(X_s)|$ is arbitrarily large.

Edit: (This is an expansion of the argument I sketched in the comments.) Note that for any stochastic proces $(X_t)=(X_t)_{0 \leq t \leq 1}$ if $U$ is a uniform $[0,1]$ random variable, independent of $(X_t)$, then the process $(X_t')$ obtained from $(X_t)$ by cyclically shifting $(X_t)$ by $U$, has cyclically exchangeable increments. Furthermore, if $(X_t)$ itself has cyclically exchangeable increments, then $(X_t)$ and $(X_t')$ have the same distribution.

Now let $(Z_s)=(Z_s)_{0 \leq s \leq 1}$ be a Lévy process. Let $U$ be uniform on $[0,1]$ and independent of $(Z_s)$, and let $(Y_s)=(Y_s)_{0 \leq s \leq 1}$ be the process you get by cyclically shifting $(Z_s)$ by U. Then $Y_1=Z_1$, and $(Y_s)$ has the same distribution as $(Z_s)$.

Conditional upon $Z_1$ (which equals $Y_1$), we don't automatically know the distribution of $(Z_s)$. However, we know the following facts.

1. Conditional on $Z_1$, $(Y_s)$ is distributed as a uniformly random cyclic shift of the conditioned process $(Z_s)$ (conditioned on $Z_1$), so still has has cyclically exchangeable increments.

2. Since $Z_1=Y_1$, $(Y_s)$ conditioned on $Z_1$ is the same as $(Y_s)$ conditioned on $Y_1$. But $(Y_s)$ and $(Z_s)$ have the same distribution so $(Y_s)$ conditioned on $Y_1$ is distributed as $(Z_s)$ conditioned on $Z_1$.

Putting these facts together, we see that conditional on $Z_1$, $(Z_s)$ still has cyclically exchangeable increments, and thus (still conditional on $Z_1$) $(Z_s - sZ_1)$ does as well. But then $(Z_s - sZ_1)$ is a process with c.e. increments and equal to zero at $s=0$, $s=1$. By the first three paragraphs of my answer, it follows that if $\sup_{0 \leq s \leq 1} |\mathbb{E}(Z_s -sZ_1 | Z_1)|$ is almost surely finite, then almost surely $\mathbb{E}(Z_s|Z_1)=sZ_1$.

But $\mathbb{E}(Z_s -sZ_1 | Z_1) = \mathbb{E}(Z_s|Z_1)+sZ_1$ so the requirement boils down to $\sup_{0 \leq s \leq 1} |\mathbb{E}(Z_s|Z_1)|$ almost surely finite. Using the tower law, this holds as long as $\mathbb{E}(\sup_{0 \leq s \leq 1} |Z_s|)$ is almost surely finite. I think this is equivalent to requiring that $\mathbb{E}|Z_1| < \infty$ but I still haven't checked.

-
Thank you Louigi. I'll go over your answer later today, or tomorrow. General comment: since the result is very useful yet apparently not well-known, a rigorous proof is desirable. –  Shai Covo Nov 24 '10 at 17:18
I think what I described is a rigorous proof. Write $\chi_s(z) = \mathbb{E}(e^{izX_s})$. By infinite divisibility of $X_t$, for rational $q \in (0,1)$ and $s=qt$, $\chi_s(z)=(\chi_t(z))^{q}$ so $\mathbb{E}(X_{qt})=0$. By bounded convergence, the same equality follows for real $r \in (0,1)$. It then follows immediately that $\mathbb{E}(X_{rt}) = \lim_{q \to r} \mathbb{E}(X_{qt}) = 0$. This works for either Ori's or my argument. –  Louigi Addario-Berry Nov 24 '10 at 18:00
Incidentally, Kai Lai Chung's "Course in probability theory" has a very careful exposition of characteristic functions and infinitely divisible distributions and in particular does all the required complex-analytic details related to choosing a branch of complex log, etcetera. –  Louigi Addario-Berry Nov 24 '10 at 18:03
For my last paragraph I should have said that all that holds conditional upon $Z_t$, or else it doesn't address your question. –  Louigi Addario-Berry Nov 24 '10 at 18:19
How can you prove that "all that holds conditional upon $Z_t$"? –  Shai Covo Nov 25 '10 at 10:58

First of all, considering the responses from this site, it seems that this result is not well-known (even among specialists), though very useful and relatively easy to derive. So, it was worth posting this here, and it is worth considering this a little further.

I'll begin with Didier's answer, which corresponds to the characteristic functions formulation (original question). The main point, using Didier's notation, is that $(a+b){\rm E}(X|S) = a S$ (what we want to show) is implied by $(a + b){\rm E}(X{\rm e}^{{\rm i}tS} ) = a{\rm E}(S{\rm e}^{{\rm i}tS} )$ for every $t \in \mathbb{R}$. Indeed, the latter condition implies $(a + b){\rm E}(X \mathbf{1}_A ) = a{\rm E}(S \mathbf{1}_A )$ for any $A \in \sigma(S)$, and thus, from the definition of conditional expectation, $(a+b){\rm E}(X|S) = a S$. Now, as Didier described, showing that $(a + b){\rm E}(X{\rm e}^{{\rm i}tS} ) = a{\rm E}(S{\rm e}^{{\rm i}tS} )$ is very easy, under the assumption ${\rm E}({\rm e}^{{\rm i}tX}) = {\rm e}^{a \psi(t)}$ and ${\rm E}({\rm e}^{{\rm i}tY}) = {\rm e}^{b \psi(t)}$. For completeness, the following point(s) should be noted here. $\frac{{\rm d}}{{{\rm d}t}}{\rm E}({\rm e}^{{\rm i}tX} ) = {\rm i}{\rm E}(X{\rm e}^{{\rm i}tX} )$ by virtue of the dominated convergence theorem (since $X$ is integrable; the same goes with respect to $Y$). So, ${\rm e}^{a \psi(t)}$ is differentiable, and from the fact that $\psi$ is continuous it follows that $\frac{{\rm d}}{{{\rm d}t}} {\rm e}^{a \psi(t)} = {\rm e}^{a \psi(t)} a \psi'(t)$, which we needed for the proof. [Interestingly, this shows that if $X$ is an integrable ID rv, then the corresponding characteristic exponent, $\psi$, is differentiable.] So overall, it seems that Didier indeed provided a rigorous but (relatively) simple proof.

Ori's answer, on the other hand, corresponds to the L\'evy process formulation. My original proof of the result completes Ori's answer (the beginning is essentially the same). Here it is. Suppose that $X$ is an integrable L\'evy process, and fix $0 < s < t$. Assume first that $s/t=m/n$, with $m,n \in \mathbb{N}$. From $\sum\nolimits_{i = 1}^n {{\rm E}[X_{it/n} - X_{(i - 1)t/n} |X_t ]} = X_t$ we deduce that ${\rm E}[X_{t/n}|X_t]=X_t / n$, and, in turn, ${\rm E}[X_s |X_t ] = (m/n)X_t = (s/t)X_t$. If $s/t$ is irrational, let $(s_j)$ be a sequence such that $s_j \uparrow s$ with $s_j/t$ being rational. By an elementary property of L\'evy processes, $X_{s_j } \stackrel{{\rm a.s.}}{\rightarrow} X_s$. Define $X_s^* = \sup _{u \in [0,s]} |X_u |$; thus $|X_{s_j}|\leq X_s^*$ $\forall j$. Since, by assumption, ${\rm E}[|X_s|]<\infty$, we conclude from Theorem 25.18 in the classical book "L\'evy Processes and Infinitely Divisible Distributions" (by Sato) that also ${\rm E}[X_s^*]<\infty$. Hence, by the dominated convergence theorem for conditional expectations, ${\rm E}[X_{s_j } |X_t ] \stackrel{{\rm a.s.}}{\rightarrow} {\rm E}[X_s |X_t ]$. Since $s_j/t$ is rational, ${\rm E}[X_{s_j } |X_t ]=(s_j/t)X_t$. Thus, ${\rm E}[X_s |X_t ] = (s/t)X_t$.

Finally, Louigi's approach may be useful in a more general setting. In this context, I find it interesting to consider ${\rm E}(X_s | X_t)$ ($0 < s < t$) for general processes (cf. its counterpart ${\rm E}(X_t | X_s)$). Any ideas?

-
At the risk of running against the tide of this (very interesting) MO page, I must confess being less and less convinced by the infinite divisibility (ID) aspect of the problem. (For instance, re a remark in Shai's last post above, the characteristic exponent of EVERY integrable random variable, ID or not, is differentiable.) To wit: the proof I explained does not use ID; it works for every distribution mentioned here (normal, Poisson, Gamma, integrable Lévy); it works also for distributions which are far from ID. (But cyclic exchangeability is a nice tool.) –  Did Dec 8 '10 at 12:46
Thank you for this insightful comment. So, this leaves space for further study. I have some idea, which I'll probably present later. –  Shai Covo Dec 8 '10 at 14:19

The following was motivated by Didier's comment given below my first answer.

On the one hand, the role of infinite divisibility (ID) might not seem important in our context, in view of the following general example (and, moreover, part of the next paragraph). If $Z$ is any integrable random variable, and if $a/(a+b)$ is rational, say $a/(a+b)=n_1/(n_1+n_2)$ with $n_1,n_2 \in \mathbb{N}$, then letting $X = \sum\nolimits_{i = 1}^{n_1 } {Z_i }$ and $Y = \sum\nolimits_{i = n_1+1}^{n_1+n_2 } {Z_i }$, where $Z_i$ are independent copies of $Z$, we have ${\rm E}( X|X + Y)=\frac{a}{{a + b}}(X + Y)$. As a side note, it is worth noting here that for $X \sim {\rm binomial}(n_1,p)$, $Y \sim {\rm binomial}(n_2,p)$ this gives ${\rm E}(X|X+Y)=\frac{{n_1 }}{{n_1 + n_2 }}(X+Y)$, a result which might be quite difficult to obtain directly, that is by calculating $\sum\nolimits_{k = 0}^{n_1 } {k{\rm P}(X = k|X + Y = n)}$ (this can be a challenging exercise for students).

On the other hand, consider the following question. Suppose that $X$ and $Y$ are independent integrable random variables with characteristic functions $\varphi_X$ and $\varphi_Y$, respectively, and $a$ and $b$ are positive real constants. Is it true that ${\rm E}(X|X+Y) = \frac{a}{{a + b}}(X + Y)$ if and only if $\varphi_Y = \varphi_X^{b/a}$? If $X$ is ID, then the condition $\varphi_Y = \varphi_X^{b/a}$ implies ${\rm E}(X|X+Y) = \frac{a}{{a + b}}(X + Y)$. Didier's answer suggests that this is true in general, and moreover that the opposite implication might be true as well, since it gives rise to the differential equation $b\frac{{\varphi'_X }}{{\varphi _X }} = a\frac{{\varphi' _Y }}{{\varphi _Y }}$, hence to $\varphi_Y = \varphi_X^{b/a}$ (note that $\varphi _X (0) = \varphi _Y (0) = 1$). It might be important to point out here that the characteristic function of an ID random variable has no zero. However, if $X$ is not ID, then $\varphi_X^{b/a}$ might not be a characteristic function. Indeed, if $\varphi {}_X^c$ is a characteristic function for all $c>0$, then from $\varphi _X = (\varphi _X^{1/n} )^n$ $\forall n$ it would follow that $X$ is ID. So, it seems that infinite divisibility does play an important role in our context.

Finally, do you think that indeed ${\rm E}(X|X+Y) = \frac{a}{{a + b}}(X + Y)$ if and only if $\varphi_Y = \varphi_X^{b/a}$? It is quite an important result, if it is true...

-
About your "On the one hand" paragraph: indeed, to compute conditional distributions is often a bad idea (which I would not call "direct") if one is interested in a conditional expectation. In the case at hand, the exchangeability of the sequence $(Z_i)$ shows that $E(Z_i|X+Y)$ does not depend on $i$, and $E(X|X+Y)$ follows. –  Did Dec 28 '10 at 11:34
At least it leads to interesting identities, which might even be quite difficult to prove (e.g. in the binomial case). –  Shai Covo Dec 28 '10 at 11:43
About your "On the other hand" paragraph: indeed, $(a+b)E(X|X+Y)=(a+b)(X+Y)$ if and only if $b\varphi'_X\varphi_Y=a\varphi'_Y\varphi_X$. And this last condition implies that (the principal determinations of) the complex valued functions $\varphi_X^b$ and $\varphi_Y^a$ coincide in a neighborhood of $0$. But, in general, I simply do not know what it is you call $\varphi_X^{b/a}$. Finally, note that, even assuming that all the functions $\varphi_X^c$ are well defined, the condition that $\varphi_X^c$ be a characteristic function for every $c>0$ is nowhere in the original question. –  Did Dec 28 '10 at 11:46
Note the typo in the first line (coefficient of $(X+Y)$). –  Shai Covo Dec 28 '10 at 12:39
In some respect, the problem is particularly suitable for ID variables, since $\varphi_X^{b/a}$ is well-defined for any $a,b>0$ if $X$ is ID. So, the question at the end of my new answer can be answered in the ID setting. –  Shai Covo Dec 28 '10 at 12:53