# exchangeable normal r.v.s

Usually "exchangeable normal random variables" means jointly normal random variables $X_1,\ldots,X_n$ (i.e. so distributed that every linear combination of them is normally distributed) that are exchangeable in the sense that if no matter you permute the indices, you don't alter the probability distribution of the $n$-tuple.

But I wonder about a different sort of exchangeable normals. Is there some probability distribution of an infinite sequence $X_0,X_1,X_2,\ldots$ of random variables such that

$\bullet$ For each $i$, $X_i \sim N(0,1)$ (marginal normality),

$\bullet$ Finite permutations of the indices never alter the probability distribution of the sequence as a whole (exchangeability),

$\bullet$ For any Borel set $A$, $\lim\limits_{n\to\infty}\Pr(X_0\in A \mid X_1,\ldots,X_n) = \lim\limits_{n\to\infty} \dfrac{|A\cap \lbrace X_1,\ldots,X_n \rbrace|}{n}$

$\bullet$ Any reasonably well behaved distribution could be the limiting distribution; which one it is would determine the nature of the dependence among $X_1,X_2,X_3,\ldots.$

?

(If I'm not mistaken, it would be enough to show the third bullet point is satisfied whenever $A$ is a half-infinite interval.)

Later edit: At least initially, I'm leaning toward construing "reasonably well behaved" as meaning having an everywhere strictly positive density with respect to Lebesgue measure (or---what is the same thing---a strictly positive density with respect to the marginal distribution). And I wonder if the answer might changed if we required it to have the same mean and variance as the marginal distribution. I'm thinking of the limiting distribution as a "population distribution", about whose nature one becomes less uncertain as the sample size grows.

Still later edit: I'm leaning toward construing the fourth bullet point something like this: You pick any not too nasty absolutely continuous probability distribution. Pick an infinite i.i.d. sample from it. Feed that sample into what you see above, i.e. find $\lim\limits_{n\to\infty} \Pr(X_0\in A\mid X_1=x_1, X_2 = x_2, \ldots, X_n=x_n)$ where $x_1,x_2,x_3,\ldots$ is what you got. Then the third bullet point should be satisfied. Then question is: can we guarantee that by some judicious choice of the joint probability distribution right at the outset, also satisfying the other bullet points?

-
This place really needs to upgrade its software to the sort of thing used on stackexchange. I had no idea that typing \{ and \} within TeX didn't work here. –  Michael Hardy Sep 7 '11 at 19:28
.....and neither does \left\{ \right\}. –  Michael Hardy Sep 7 '11 at 19:29
I finally got it to work by using \lbrace and \rbrace. –  Michael Hardy Sep 7 '11 at 19:35
@Michael you can also do \\{ and \\} (I've found). The first backslash seems to be treated as an escape character before interpreting the TeX, so that \\ sends \ to the TeX interpreter, and \\{ sends \{ as desired. But I don't know why. –  David M Kaplan Sep 7 '11 at 19:46
@David: Thank you. –  Michael Hardy Sep 7 '11 at 19:49

I will expand on this answer later if there is interest and when I have some references handy. But for now you may be interested in the following way of thinking about the problem.

One way to characterize the Gaussian distribution is as the unique distribution on $\mathbb{R}$ satisfying spherical symmetry. More precisely, for $N$ observations, consider the two-dimensional statistic $$T(X_{1:N}) = \left(\sum_{i=1}^N X_i, \sum_{i=1}^N X_j^2 \right).$$ Assume that the conditional distribution of the vector $X_{1:N}$ is uniform on the hypersphere with center $(T_1, \dots, T_1)$ and radius $\sqrt{(T_2 - T_1^2/N)}$. This implies that the density for $X_{1:N}$ may be written as $$f(X_{1:N}) = \int \prod_{j=1}^N \left[(2\pi)^{-\frac{1}{2}}\sigma^{-1} \exp{\left\lbrace\frac{(x_j - \mu)^2}{\sigma^2}\right\rbrace}\right] dP(\sigma, \mu)$$ for some density $dP(\sigma,\mu)$. (This sort of representation theorem motivates the use of prior distributions in Bayesian statistics.) Note that as $N \rightarrow \infty$, $T$ converges to the true first and second moments, which gives back another common characterization of the normal distribution as being specifiable using only them.

So, when you say that "usually 'exchangeable normal random variables' means jointly normal random variables" it makes me wonder what is the more critical property, the permutation invariance -- which does not uniquely define the distribution -- or the underlying symmetry -- which in the case of the normal distribution does. The reason I brought up the copula earlier is that I think getting your necessary marginals to be whatever is not much of a barrier, because you can always transform things elementwise. This makes me think that you are really asking whether or not there are other forms of exchangeable distributions of real-valued vectors, and there definitely are. Following the example of the spherical symmetry, the basic recipe is to specify a statistic and then specify a uniform transition kernel given the value of that statistic. This approach has been systematized by Steffen Lauritzen in a monograph S. L. Lauritzen. Extremal Families and Systems of Sufficient Statistics. Lecture Notes in Statistics, No. 49. A good textbook treatment of this is given in section 2.4 of Mark Schervish's Theory of Statistics (available on Google Books, but my toolbar for providing links seems to have vanished).

Apologies if you knew all of the above and I missed the point of your question, but your comment to Yuri makes me think that this stuff would be of interest. The keywords you'd want to include to dig around more include "de Finetti theorems", "extremal families", and "partial exchangeability".

-
I knew some of this, and in other parts I'm not sure yet what you're saying (just read it a minute ago). –  Michael Hardy Sep 8 '11 at 2:29
What is a "transfer kernel"? Does that mean the conditional probability distribution of the data given the value of the particular statistic? –  Michael Hardy Sep 8 '11 at 3:19
Mainly I just wanted to advertise the work of Lauritzen (summarized in the section of Schervish that I mention) because it is really beautiful and seems directly relevant. It isn't often that I have the opportunity to bring it up. The missing link is that that work does not talk about prescribing marginal distributions; my thought was that this may be handled using a copula-like construction, but that part is definitely half baked! –  R Hahn Sep 8 '11 at 3:23
Sorry, the correct term is "transition kernel", and it acts like a conditional probability distribution effectively in this case. –  R Hahn Sep 8 '11 at 3:28
books.google.com/… –  R Hahn Sep 8 '11 at 3:39

De Finetti's theorem has already been mentioned, but it seems to me that it answers the original question. In this case, it says that any exchangeable infinite sequence $X_1, X_2, X_3, \ldots$ of real-valued random variables comes from some probability measure $\Phi$ on the set of measures on $\Bbb R$. The sequence is generated by picking $\mu\sim\Phi$ and then taking i.i.d. $X_1, X_2, X_3, \ldots \sim\mu$. So, the third bullet point is automatically satisfied, and the "population distribution" is $\mu$. To get marginal normality you only need that ${\bf E}[\mu]=N(0,1)$, so there is a wide choice for $\Phi$.

-
I think you may be right..... I'm wondering if I need to figure out how to get a handle on which distributions $\Phi$ have expectation $N(0,1)$. –  Michael Hardy Sep 9 '11 at 23:06
Lauritzen gives a simple example in his lecture. He fixes some $\rho$ in $[0,1]$ and then takes $\mu=N(Y,1-\rho)$, where $Y\sim N(0,\rho)$. This interpolates between the i.i.d. case ($\rho=0$) and the case where $X_1$, $X_2$, ... is a constant sequence ($\rho=1$.) –  David Moews Sep 9 '11 at 23:52
Another example, in which $\Phi$ is discrete, would be to draw the graph of the density function of $N(0,1)$, divide the region underneath the density function into countably many chunks, and pick $\mu$ by throwing a dart at the picture and looking at the chunk it lands in. To pick the $X_i$'s, we throw more darts at the picture, but condition them to land in the chunk we chose earlier. –  David Moews Sep 9 '11 at 23:55
Maybe my fourth bullet point isn't clear, and maybe it can't be made clear without a lot more work. Lauritzen's example goes only a tiny distance toward what I had in mind. The distribution given by $\lim\limits_{n\to\infty} \dfrac{|A\cap \lbrace X_1,\ldots,X_n \rbrace|}{n}$ would always be normal. Not the same normal every time, but always normal. And the sequence actually is multivariate normal in the sense that every linear combination of the $X$s is normal. (And the reason why he used the lower-case letter $\rho$ for that parameter seems to be just what you'd guess.) –  Michael Hardy Sep 11 '11 at 20:39
True, Lauritzen's example is multivariate normal. OK, here's another example: pick $\alpha\sim N(0,1)$, and then let $X_i$ be i.i.d., each with density $2\phi(x) \Phi(\alpha x)$, where $\phi$ and $\Phi$ are the density and cdf of $N(0,1)$. In this example the $X_i$'s have skewed distributions where $\alpha$, a measure of the skew, is itself normally distributed. –  David Moews Sep 12 '11 at 8:13

i.i.d. standard normals seem to work, don't they?

-
i.i.d. standard normals don't satisfy the fourth bullet point. –  Michael Hardy Sep 7 '11 at 23:20
'nother words, I'd like to be able to prescribe the limiting distribution and then set up the nature of the dependence among the random variables so that that's what it converges to. –  Michael Hardy Sep 7 '11 at 23:22
@Michael Hardy: I see. The words "any" and "could" in your point 4 are misleading. Btw, Halmos in his "How to write mathematics" advises against using "any" in mathematical writing. –  Yuri Bakhtin Sep 8 '11 at 22:33
I never write "any" in such contexts as "If any A is B, then....". That's ambiguous since it could mean "If it is the case that any A, no matter which one, is B, then...." or it could mean "If there is any A that is B, then....", which has quite a different meaning. I'll look at rephrasing what I wrote here. –  Michael Hardy Sep 9 '11 at 18:41