Timeline for Singular Fisher information matrix and existence of unbiased estimators
Current License: CC BY-SA 4.0
28 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 4, 2023 at 1:44 | vote | accept | JNL | ||
| Jul 4, 2023 at 1:10 | answer | added | Iosif Pinelis | timeline score: 2 | |
| Jul 3, 2023 at 23:33 | comment | added | JNL | And further, at $(x,0)$, the y-coordinate MLE is asymptotically unbiased, even though the Fisher information matrix is singular at $(x,0)$. I think you have provided the answer for that in your previous comment that it is possible for MLE to have zero bias at parameter values $\theta_0$ where $I(\theta_0)$ is singular. I don't think I found any reference/proof for this though when I was searching for it. Can you please share a ref for that if you know one? | |
| Jul 3, 2023 at 23:13 | comment | added | JNL | @IosifPinelis, thank you for your comments. I apologise for the lack of specificity for many of the points you raised, I'm an engineer so I have much less experience of defining them in that way - I will try to do that as much as I can. About the collinear sensors in my question: I understand that for $y\ne0$, the model is non-identifiable. But, what I find unusual is that for $y\ne0$, the Fisher information matrix is full rank, but at the only $y$-coordinate value where the model is identifiable (i.e. $y=0$), the Fisher information matrix is singular. | |
| Jul 3, 2023 at 14:45 | comment | added | Iosif Pinelis | As for your particular example, with the three sensors placed (for a strange reason) collinearly, it is clear that they cannot detect the sign of $y$. More formally, that statistical model is not identifiable, since the model distribution for $(x,-y)$ is the same as that for $(x,y)$. So, it is impossible to consistently or unbiasedly estimate $y$ in that model if $y\ne0$. | |
| Jul 3, 2023 at 14:31 | comment | added | Iosif Pinelis | On the other hand, if your question is "Can the MLE have zero bias at a parameter value $\theta_0$ such that the Fisher information matrix $I(\theta)$ is singular at $\theta=\theta_0$?", then the answer is "yes, of course". | |
| Jul 3, 2023 at 14:09 | comment | added | Iosif Pinelis | Also, avoid using pronouns like this, that, which -- because it is often unclear what they refer to. | |
| Jul 3, 2023 at 14:07 | comment | added | Iosif Pinelis | Also, it is unclear what you mean by "unbiased". In standard terminology, an estimator is unbiased if its bias is $0$ for all values of the parameter. Maximum likelihood estimators (MLEs) are rather rarely unbiased. For instance, the MLE of the variance for an iid normal sample with unknown mean and variance is biased. | |
| Jul 3, 2023 at 13:34 | comment | added | Iosif Pinelis | I asked what your plots are, not what they show or what they are for. Things can be shown in many ways and many things may be for something. Plots can and should be described formally in mathematical terms, such as sets and functions. E.g. a plot can be formally described as the set $\{(x,\sin x)\colon 0\le x\le\pi\}$. I would like to see descriptions like this. | |
| Jul 3, 2023 at 10:23 | comment | added | JNL | (cont. from prev. comment). The 1st plot's left and right graphs are for two different values of the random measurement noise (i.e. two different randomly generated $\epsilon$ vectors). The 2nd plot shows maximum likelihood estimates (blue points) of $\theta$ for a simulation of 1500 random measurement noise vectors. These simulation results seem to suggest that the $y$-coordinate MLE is unbiased, but the Fisher information matrix at that point is singular. So my question is whether that is possible, and if not where I might have gone wrong. | |
| Jul 3, 2023 at 10:22 | comment | added | JNL | @IosifPinelis, if I were to try and frame the question more concretely, I think it would be: is it possible for an unbiased estimator to exist when the Fisher information matrix is singular? Because I think this is what I am seeing. As for the plots: the 1st plot shows the negative log-likelihood (i.e. $−L(x,y)$ which is given in the question), and the circles are the $\theta=(x,y)^T$ points that satisfy $r_j=\|\theta−\theta_j\|_{2}+\epsilon_j$ where $r_j$ is the measured range from the $j^{th}$ sensor situated at $\theta_{j}$. | |
| Jul 3, 2023 at 2:37 | comment | added | Iosif Pinelis | What is the question here, specifically? What are the plots, specifically (formally) described? What are the 'circles"? ... | |
| Jun 23, 2023 at 1:27 | comment | added | JNL | Thanks @πr8, I think I understand the concept of the reparamatrisation now. In this case, I might have to consider how to reparametrise the FIM. Do you have any reference for the Bernoulli example you gave? I tried looking it up, but didn't find anything very helpful. Also, I have another question: you mentioned that when the FIM is non-singular, one expects the MLE to approach MLE~(True_param+Gaussian_noise). I was wondering if you know whether FIM non-singularity is a sufficient condition for this? I.e. is it possible for a parameter to be non-identifiable but the FIM to be non-singular? | |
| Jun 15, 2023 at 10:35 | comment | added | πr8 | I was a bit unclear, sorry. When the FIM is non-singular, one expects that the MLE asymptotically behaves like MLE ~ True_Parameter + Approximately_Gaussian_Noise. Your plots seem to suggest that this is not happening in your case (which is natural, due to the singularity of the FIM). However, it looks plausible that instead MLE ~ Nonlinear_Function( True_Parameter + Approximately_Gaussian_Noise), or even that Nonlinear_Function(MLE) ~ Nonlinear_Function(True_Parameter) + Approximately_Gaussian_Noise. The reparametrisation idea relates to this last point; I will try to write something soon. | |
| Jun 14, 2023 at 13:42 | history | edited | JNL | CC BY-SA 4.0 |
deleted 24 characters in body
|
| Jun 14, 2023 at 13:38 | comment | added | JNL | Thanks for your suggestions in improving the question @πr8. I hope it is clearer now with the definitions of $\phi$ and the likelihood function. Sorry, I'm not sure I understand the reparameterisation you mentioned: do you mean that the Gaussian FIM would not apply in this case? The noise I am adding in the simulations is Gaussian, and I think the two situations that arise from the intersecting and non-intersecting circles show that the estimates would lie along y=0. Is there a condition on the applicability of the FIM or CRB that I might be missing maybe? | |
| Jun 14, 2023 at 13:10 | history | edited | JNL | CC BY-SA 4.0 |
Typo
|
| Jun 14, 2023 at 13:04 | history | edited | JNL | CC BY-SA 4.0 |
Added figure for phi and wrote out NLL function explicitly; added 9 characters in body
|
| Jun 12, 2023 at 10:20 | comment | added | πr8 | i) could you possibly define the $\phi_j$ for clarity?, ii) could you possibly write down the likelihood explicitly?, iii) a cursory thought is that it might be that the FIM is singular in terms of your given parametrisation, but non-singular under a simple reparametrisation, e.g. for estimating the success probability of a Bernoulli random variable, the FIM for $p$ becomes singular at $0$ and $1$, but there is a simple function $f$ such that the FIM for $f(p)$ is constant. the non-Gaussianity of your estimates suggests the MLE may be close in law to a nonlinear transformation of a Gaussian. | |
| Jun 12, 2023 at 4:34 | history | edited | Michael Hardy | CC BY-SA 4.0 |
added 8 characters in body
|
| Jun 12, 2023 at 4:15 | history | edited | Michael Hardy | CC BY-SA 4.0 |
added 15 characters in body
|
| Jun 12, 2023 at 4:08 | history | edited | Michael Hardy | CC BY-SA 4.0 |
deleted 78 characters in body
|
| Jun 10, 2023 at 6:56 | history | edited | YCor | CC BY-SA 4.0 |
removed capitals from title
|
| Jun 10, 2023 at 2:16 | history | edited | JNL | CC BY-SA 4.0 |
Corrected some spellings
|
| Jun 10, 2023 at 2:12 | history | edited | JNL | CC BY-SA 4.0 |
added 3 characters in body
|
| Jun 10, 2023 at 2:08 | history | edited | JNL | CC BY-SA 4.0 |
Bit more background on why I'm posting here
|
| S Jun 10, 2023 at 2:06 | review | First questions | |||
| Jun 10, 2023 at 2:42 | |||||
| S Jun 10, 2023 at 2:06 | history | asked | JNL | CC BY-SA 4.0 |