Find a way to apply the MLE on Fisher or Covariance matrix to make cross-correlations

Question

I have 2 Fisher matrixes which represent information for the same variables (I mean columns/rows represent the same parameters in the 2 matrixes).

Now I would like to make the cross-correlations synthesis of these 2 matrixes by applying for each parameter the well known formula (coming from Maximum Likelihood Estimator method) :

$$\dfrac{1}{\sigma_{\hat{\tau}}^{2}}=\dfrac{1}{\sigma_1^2}+\dfrac{1}{\sigma_2^2}\quad(1)$$

$\sigma_{\hat{\tau}}$ represents the best estimator representing the combination of a sample1 ($\sigma_1$) and a sample2 ($\sigma_2$).

Now, I would like to do the same but for my 2 Fisher matrixes, i.e from a matricial point of view.

1. Firstly, for this, I tried to diagonalize in a simultaneous way each of these 2 Fisher matrix, i.e by finding a common eigenvectors basis for both. Then, I would add the 2 diagonal matrices and I have so a global diagonal Fisher matrix, and after come back in the space of start.

But this method gives the same constraints (by inversing the Fisher matrix) are the same than a classical synthesis between 2 matrices since :

$V^{-1} A V = D_1$

$V^{-1} B V = D_2$ ,

then what I wanted to do by summing the 2 diagonal matrices $D_1$ and $D_2$ is the same thing than doing $A+B$ :

$A+B = V (D_1+ D_2) V^{-1}$

Finally, I had to give up this track

2. Secondly, I tried to work directly in the space of Covariance matrix. I diagonalized "simultaneously" each of the 2 matrices. (Then, I have no more covariances terms).

and I build another covariance matrix by applying the MLE like putting on the diagonal :

$$\sigma_{\hat{\tau}}^{2}=\dfrac{1}{\dfrac{1}{\sigma_1^2}+\dfrac{1}{\sigma_2^2}}\quad(2)$$

Unfortunately, it doesn't increase the FoM (Figure of Merit, equal to $\dfrac{1}{\sqrt{\text{det(block(2 parameters))}}}$ ), I mean that constraints are not better : as a conclusion, I can't manage to do cross-correlations since I have no gain on constraints.

Moreover, with the first method 1) above, after the building of Fisher matrix and its inversion, I can marginalize over the nuisance parameters and re-invert to get a Fisher matrix where nuisance parameters estimations are encoded into it.

But I can't do the same for method 2). I can fix parameters (remove directly lines/columns) in Fisher matrix but this produces too high FoM (and so too small constraints) since I have less error parameters to estimate.

So, my question is about the research of a way to apply the MLE on Fisher (or maybe directly on Covariance matrix) to make cross-correlations between 2 given Fisher matrices (which represent actually 2 different probes into a cosmology context).

Any suggestion/track/clue about this way to do is welcome.

UPDATE 1: @fedja . Thanks a lot for your support and answer. I have a good and bad news. Firstly, it seems that you have formulated my issue on a pure mathematical point of view and when you finally find that :

$$Fisher_{\text{total}}=\left(\sum_j S_j^{-1}\right)\quad(1)$$

This makes sense with the classical combination between 2 Fisher matrix. Indeed, If I take $S_j^{-1}$ as a Fisher matrix (2 matrices actually by taking $j=1,2$), I find the classical synthesis of final covariance coming from summing all the Fisher matrices ( $\left(\sum_j S_j^{-1}\right)$) and finally inverse this sum of the Fisher matrices to get the final covariance matrix by doing :

$$S=\left(\sum_j S_j^{-1}\right)^{-1}\quad(2)$$

By the way, could you give me please the details to find the below expression $(3)$ which is appropriate when minimizing $S=\sum_j U_jS_jU_j^*$ :

$$U_j=(\sum_j S_j^{-1})^{-1}S_j^{-1}\quad(3)$$

The bad news is that I think I can't get further in cross-correlations between the 2 Fisher matrix from a mathematical point of view.

I have now to put more physics hypothesis. For example, here the formula in cosmology to compute the cross-correlated Fisher matrix for 2D (photometric and Weak lensing): an element $(\alpha,\beta)$ of this matrix is equal to :

By posting this message here, I thought that I could combine relatively easily combine the spectroscopic probe (3D) with 2D such that I would a final synthesis 2D+3D for my final Fisher matrix and so the fulll covariance matrix of the synthesis 2D+3D.

But unfortunately, I have to give up the hope of handling the case 2D+3D with only mathematics.

UPDATE 2: Just a last question : what do you think about pooled variance. I would like to find a relevant $b$ parameter in the expression :

$$s_{\text{total}} = bs_1 + (1-b)s_2 ; V(s_p)=b^2 V(s_1) + (1-b)^2 V(s_2)$$

where $V(s_1)$ and $V(s_2)$ represent the standard variance of my 2 samples, so the method called "Pooled variance" : here an example

Could you think it could help me by fixing this value to 0.5 (the first Fisher matrix brings same information as the second one) ? It would deserves another post on maths forum, doesn't it ?

Answer 1

OK, let's turn the tables around. I'll describe in my language a physical setup, the mathematical formulation, and the answer that fit the three words I understand: "detector", "error", and "formula (1)" and you'll try to figure out whether they have anything to do with your problem :-)

Physical setup: You have an object whose true position $x$ is represented by a point in $\mathbb R^n$ and $m$ detectors that read it as $x_j=x+y_j$ ($j=1,\dots,m$) where $y_j$ is mean $0$ Gaussian noise with known correlation matrix $S_j=Ey_jy_j^*$. Noises for different detectors are independent and $S_j$ may be quite different. How to combine $x_j$ to get the least square error in every direction simultaneously?

Mathematical formulation: Put $\hat x=\sum_j U_jx_j$ where $U_j$ are arbitrary linear operators with $\sum U_j=I$ (to have an unbiased estimator). We want to minimize the covariance matrix $S=\sum_j U_jS_jU_j^*$ of $\hat x-x$ in the sense of positive definite quadratic forms.

Solution: (well-known, of course, but I'll provide the details if needed) Put $U_j=(\sum_j S_j^{-1})^{-1}S_j^{-1}$ (usual matrix inversion and multiplication) and get $$ S=\left(\sum_j S_j^{-1}\right)^{-1}\tag 1 $$ which is optimal in the sense that for any other choice of $U_j$, you'll get a quadratic form dominating this one.

Now let me know if this setup and the final result have anything to do with what you want :-)

Edit: The details requested by the OP.

If $\hat x=\sum_j U_j x_j=\sum_j U_j(x+y_j)$ and $\sum_j U_j=I$, then $$ E(\hat x-x)(hat x-x)^*=\sum_{j,k} U_jEy_jy_k^*U_k^*=\sum_j U_jS_jS_j^* $$ because $Ey_jy_k^*=0$ if $j\ne k$ (independence) and $Ey_jy_j^*=S_j$ Hence the formula for the variance of the linear estimator.

Suppose now that we have some $U_j$ that satisfy $\sum_j U_j=I$. We want to show that $\sum_j U_jS_jU_j^*\ge\left(\sum_j S_j^{-1}\right)^{-1}$ in the sense of quadratic forms, i.e., that $$ \sum_j \langle S_jU_j^*z,U_j^*z\rangle\ge \left\langle\left(\sum_j S_j^{-1}\right)^{-1}z,z\right\rangle $$ Note that $\sum_jU_j^*=I^*=I$, so $z=\sum_jz_j$ where $z_j=U_j^*z$. Now the desired inequality is just the reformulation of the convexity property of the mapping $(W,z)\mapsto\langle W^{-1}z,z\rangle$ where $W$ is running over the cone of positive semi-definite matrices and $z$ is running over $\mathbb R^n$. The choice of $U_j$ that I gave attains that low bound.

In general it looks like a very partial case of the Rao-Cramer lower bound for the variance of an unbiased estimator (which, I suspect, it is), so it shouldn't be surprising for you.