What are some of results in low dimensional statistics that do not hold in high dimensions?

Question

This question is partially inspired by the following MO post: What are some of the surprising results of finite sample statistical estimation? and current heated research front of high dimensional statistics.

Instead of asking about surprising results in high dimensions, I will ask what kind of results that holds in low dimensions fails to hold in a higher dimension. And what is its relation with other mathematical branch?

(By high dimensional statistics we usually refer to a high dimension covariate space instead of response space. For example, in a regression setting $Y=f(X)$ we tend to say a problem is of high dimension if $\dim\mathcal{X}\gg \dim\mathcal{Y}$)

For one simplest example, we know that James-Stein estimator performs better than maximum likelihood estimator in terms of $L^2$ norm when the dimension $\dim\mathcal{X}=d\geq 3$; and it turns out to be an equivalent statement that a symmetric random walk in $\mathcal{X}=\mathbb{R}^d$ is transient when $d\geq 3$ via an infinitely divisible stochastic process. Another example is provided in the answer below.

Are there other such examples that can relate high dimensional phenomena in statistics?

Answer 1

A great example that I have in mind is the concentration phenomena in high dimensions. Consider the simplest multivariate normal distribution $X\sim N_d(0_d,I_d)$, we can compute its $L^2$ norm $\sum_iX^2_i=:\|X\|^2\sim\chi^2(d)$. and $X_i^2\sim \chi^2(1)$ independently. With central limit theorem applied on each component, we have that $$\frac{1}{d}\sum^d_{i=1}X^2_i=\frac{1}{d}\|X\|^2\overset{P}{\rightarrow}N_1(1,\frac{2}{d})$$ as $d\rightarrow\infty$. Use the delta method we can see that $\|X\|\overset{P}{\rightarrow}\sqrt{d}N_1(1,\frac{1}{d})=N_1(\sqrt{d},1)$ and therefore we can actually assert that as dimension $d=dim\mathcal{X}\rightarrow \infty$ the random vectors are concentrated around a sphere.

Even more surprising is that if we have another independent $Y\sim N_d(0_d,I_d)$, then we can compute the distribution of $\frac{X\cdot Y}{\|X\|\|Y\|}$ as $d\rightarrow \infty$ is $N_1(0,d)$(multidimensional CLT and delta method) and the distribution of $\|X-Y\|$ as $d\rightarrow \infty$ is $N_1(0,2d)$. These two results claimed that as $d\rightarrow \infty$ two random vectors are most likely to be orthogonal and evenly distributed on the sphere, which is not expected when $d=1,2$.

I really hope to know if there are more such examples with a motivation from consideration of the difference between the geometry of high and low dimensional spaces.