Central limit theorem with degenerate covariance matrix

Question

Are there known generalisations of the central limit theorem for several random variables when the covariance matrix is degenerate?

The usual proof of CLT based on characteristic functions (see e.g. Wikipedia) yields a degenerate multivariate normal distribution. I'm after any kind of result that resolves that degeneracy. Before taking the limit one evidently has a well behaved probability distribution for the sum of $N$ instances of each of the random variables, and one may anticipate that the width of the distribution of the sum in the degenerate directions is less than $\sqrt{N}$. But can more be said in general?

Answer 1

That the covariance matrix (say $\Sigma$) is degenerate just means that the corresponding random vector $X$ (say in $\mathbb{R}^k$) lies almost surely in a proper linear subspace $V$ of $\mathbb{R}^k$, say of dimension $r<k$. Replace the random vector $X$ by the $r$-tuple (say $Y$) of the coordinates of $X$ in an arbitrary basis $B$ of $V$. Then the covariance matrix of $Y$ will be nonsingular, and so, the multivariate CLT will be applicable -- to iid copies $Y^{(1)},\dots,Y^{(n)}$ of the vector $Y$ in $\mathbb{R}^r$.

The subspace $V$ can be described as the orthogonal complement to $\mathbb{R}^k$ of the null space of $\Sigma$. For the basis $B$ you can take any set of orthonormal eigenvectors (say $e_1,\dots,e_r$) of $\Sigma$ corresponding to the nonzero eigenvalues (say $\lambda_1,\dots,\lambda_r$) of $\Sigma$. Then $X=\sum_{j=1}^r Y_je_j$ and $Y=(Y_1,\dots,Y_r)$, with $Y_j=\langle e_j,X\rangle$, where $\langle\cdot,\cdot\rangle$ denotes the inner product in $\mathbb{R}^k$. The (nonsingular) covariance matrix of $Y$ will then be the diagonal matrix with $\lambda_1,\dots,\lambda_r$ on its diagonal.

Answer 2

I had to be guessing the meaning you intended for some of the terms you used in stating your question. In particular, it was unclear what you meant by "the central limit theorem" and "the covariance matrix". I then assumed you meant the CLT for iid copies, say $X^{(1)},X^{(2)},\dots$, of a random vector $X$ with (say) mean $0$ and a covariance matrix $\Sigma$, which latter may be singular. Then the CLT says that $S_n:=\sum_{j=1}^nX^{(j)}/\sqrt{n}$ converges in distribution as $n\to\infty$ to a centered Gaussian random vector $Z$ with the same covariance matrix $\Sigma$. Since the linear subspace $V$ that I introduced in the previous answer is completely determined by the covariance matrix of the random vector, this subspace is the same for each of these random vectors: $X,X^{(1)},X^{(2)},\dots,Z$. Therefore, each of these random vectors lies almost surely in this same subspace $V$, and then so does $S_n$.

If, however, you meant some non-iid version of the CLT, then indeed the subspaces $V^{(1)},V^{(2)},\dots$ corresponding to the random vectors $X^{(1)},X^{(2)},\dots$ may differ from the subspace $V$ corresponding to the random vector $Z$. However, in some sense (say in terms of the smaller eigenvalues) the subspaces $V^{(1)},V^{(2)},\dots$ should be close to $V$ "on an average", eventually. To be more specific here, one would need to know what non-iid version of the multivariate CLT you might have had in mind.

Answer 3

In this paper -- http://www.cs.cmu.edu/~odonnell/papers/prgs-ltfs.pdf -- we needed a non-iid "Berry--Esseen"-type result for vector-valued rv's with singular covariance matrix. Because of the singular covariance matrix possibility, we couldn't seem to use anything from the literature in a black-box fashion, so we had to make our own proof. We used the Lindeberg method, rather than the characteristic function method.

The final statement had a slightly unusual conclusion, with error being measured with respect to unions of orthants, because that's what our application needed. But anyway, for what it's worth, here's what we obtained:

"Let $X_1, \dots, X_n$ be independent ${\mathbb R}^d$-valued random variables satisfying $E[X_j] = 0$. Assume also that each is "$(2,4,\eta)$-hypercontractive", meaning $\|a + \eta X_j\|_4 \leq \|a + X_j\|_2$ for all $a \in {\mathbb R}^d$. (This is a 'niceness' condition, saying that the $X_j$'s don't have very skewed distributions. An example case where it holds is if $X_j$ is of the form $x_j w$, where $w \in {\mathbb R}^d$ is a nonrandom vector and $x_j$ is a random variable satisfying $\|x_j\|_4 \leq C\|x_j\|_2$; then the condition holds for $\eta = \frac{1}{\text{const}\cdot C}$.)

Let $S = X_1 + \cdots + X_n$ and write $M = Cov[S]$ (possibly singular). Assume that $M$'s diagonal entries are 1. Finally, let $G$ be the Gaussian random vector with mean 0 and covariance matrix $M$. Then $S$ is close to $G$ in the following sense: For any set $A \subseteq {\mathbb R}^d$ which is a translate of a union of orthants, $\Pr[S \in A]$ and $\Pr[G \in A]$ differ by at most poly$(d/\eta) \cdot (\sum_{j=1}^n \sigma_j^4)^{1/8}$, where $\sigma_j^2 := \|X_j\|_2^2$. (One expects this last factor to be "small" if no one of the $X_j$'s is ``too dominant'' in terms of variance.)"

As I said, the "translate of union of orthants" way of measuring is somewhat weird, but I expect that you can allow a larger class of test sets.

Answer 4

Ryan: Wouldn't such results as Theorem 15.1 in the book Normal Approximation and Asymptotic Expansions by Bhattacharya and Rao (1986) do for your purposes? It seems to provide the rate of convergence of the form $O(n^{-1/2})$, also for finite 4th moments (but without hypercontractivity assumptions), whereas your rate is $O(n^{-1/8})$, in the iid case. Also, if you don't care about the dependence on $d$, you may deal just with a translate of one orthant, which is a convex set. Then it seems you can use Th. 3.5 in the paper at http://arxiv.org/abs/1111.4073, which requires only finite 3rd moments, again for rate $O(n^{-1/2})$.

Addendum: I've just now noticed that you mentioned a difficulty concerning the possible singularity of the covariance matrix. But, as follows from my previous answer, such a difficulty is caused only by placing your random vector into too large a vector space. By confining the random vector to the right subspace, you remove the singularity, at least in the iid case.