Sometimes when doing regression analysis, we estimate our function $g(x) = E(Y |X =x )$ using an orthonormal series, and in particular we use an approximate series $g_{p_n}(x) = \sum_{k=1}^{p_n} \alpha_k e_k(x)$. In practice, we cannot expect to observe the true predictors $ X_i$, and so we observe the pairs $(Y_i, W_i)$, $W_i = X_i + U_i$ for some independent error $U_i$. This is sometimes known as classical measurement error. One might suggest estimating $\alpha_k$ by using a modified least squares regression estimator $$ \begin{align} \hat{\alpha_p} & = \arg\min_{a\in\mathbb{R}^p}\sum_{i=1}^{n}(Y_i - a \cdot E(\boldsymbol e^P(X_i)|W_i))^2f_W(W_i)^2\\ & = \arg\min_{a\in\mathbb{R}^p}\sum_{i=1}^{n}(Y_if_W(W_i) - a\cdot[f_W(W_i)E(\boldsymbol e^P(X_i)| W_i])^2, \end{align} $$

where $E(\boldsymbol e^P(X_i)|W_i) = [E(e_1(X_i)|W_i),... E(e_{p_n}(X_i)|W_i)]^\text{T}$. Naturally, our estimator should be based on what we observe $W_i$, which we call our error variable, since we do not observe the true predictors $X_i$, and the weights $f_W{W_i}$ just get rid of the denominator in the conditional expectation terms.

Then when we do use this method (for the time being let us assume we know the densities of $X$ and $W$, in practice we estimate $f_X$, $f_W$ but we assume we know $f_U$), we have that

$$ \begin{align} n^{-1}\boldsymbol{A}^{\text{T}}\boldsymbol{A} \hat\alpha_{p} = n^{-1}\boldsymbol A^\text{T}\boldsymbol Y, \end{align} $$

with $\boldsymbol Y = (Y_1,...,Y_n)^\text{T}$ and $\boldsymbol A \in \mathbb{R}^{n\times p_n}$ with $\boldsymbol A_{ij} = E(e_{j}(X_i)|W_i)$. Now what we know is that the rows of $\boldsymbol A$ are independent, but this is not the case with the columns, there are a strong dependency because the entire column is based on one single observation $W_i$.

$\textbf{QUESTION: }$ I want to get some asymptotic understanding of the smallest eigenvalue of $n^{-1}\boldsymbol{A^\text{T}}\boldsymbol{A}$, and I know there is a vast amount of literature on random matrix theory. Unfortunately most of the literature is based on iid entries of a random matrix which is what mine is not about, but there is a special area of random matrix theory which studies in detail the 'sample covariance matrix', which my matrix in interest has some resemblance of. Most of the work derived on the empirical spectral distribution leads to Marchenko - Pastur type distributions, and there is work on dependence structures, but I cannot seem to find anything on such strong dependencies as in my case. Is there any information / any ideas out there that might help tackle this scenario?


1 Answer 1


See http://www-personal.umich.edu/~romanv/papers/non-asymptotic-rmt-plain.pdf section 5.5.1, it deals with matrices of independent rows/columns

  • $\begingroup$ If I am not mistaken, 5.5.1 assumes that the columns / rows of $\boldsymbol{A}$ are isotropic, in this case I would require the rows to be isotropic, but how can this be if the row entries depend on only one observation i.e. the row entries are dependent on each other and therefore we cannot guarantee that the row vectors of $\boldsymbol{A}$ come from an isotropic distribution? $\endgroup$
    – user61038
    Commented Apr 29, 2014 at 4:01
  • $\begingroup$ Usually the methods carry over with some work (multiplying by the square-root of the covariance). See also jmlr.org/proceedings/papers/v23/rudelson12/rudelson12.pdf $\endgroup$ Commented Apr 29, 2014 at 4:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.