I am interested in covaraince matrix estimation. In brief: I have two estimates of the covariance matrix, and now I want to form a bona fide convex combination of the two.
Background: I have studied the work on regularization of covariance matrices. Assume the $R$ is the true $p\times p$ covariance matrix, and that I have $N$ observed vectors with covariance $R$. Then, a common technique, deeply studied by Ledoit and Wolf, is to regularize the estimate according to $$(1-\rho)\hat{S} + \rho I$$ where $\hat{S}$ is the sample covariance matrix. The optimal $\rho$, in a quadratic mean sense, depends on $R$ which is unknown. Therefore, a bona fide estimator is derived which is asymptotically consistent, that is as $N$ and $p$ grows, the bona fide estimator is as good as the optimal which requires knowledge of $R$.
Background of my problem: I have two sets of data, one set where N is huge but where the reliability is fairly low, and another set where N is smaller, but the accuracy per observed vector is higher than for the first set (this may seem as an utterly confusing statement, but the reason is that that in the first set, I observe $Hx+Gy$ where both $x$ and $y$ are random, and $R=HH^{H}+GG^{H}$. In the second set, I observe $Ht+Gy$, where $t$ are known training vectors so that the randomness is less.) This leaves me with two sample covariance matrices, $\hat{S}_1$ and $\hat{S}_2$, and my next step is to form a convex combination $$\hat{S} = (1-\rho)\hat{S}_1+\rho \hat{S}_2$$ I can do this, but I do not get a bona fide estimator.
Question: In the regularization framework of, e.g., LW, is it important that one shrink towards the identity? Or will the general framework hold true for a convex combination of two arbitrary matrices?
