Timeline for Derivative of log-likelihood function for Gaussian distribution with parameterized variance
Current License: CC BY-SA 4.0
14 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 3, 2023 at 0:11 | comment | added | JNL | can I please ask for your thoughts on another question I posted here on the singularity of the Fisher information matrix and the existence of unbiased estimators (mathoverflow.net/questions/448546/…)? I still haven't found an answer to this or found out where I might be going wrong. | |
| Jul 2, 2023 at 23:47 | vote | accept | JNL | ||
| Jul 2, 2023 at 23:47 | comment | added | JNL | ah! I understand now. The derivative will only be zero at the max likelihood estimate (well, that is how we calculate the estimate anyway, by finding the point where the derivative is zero), and this estimate is expected to converge to the parameter value asymptotically. My mistake in understanding was that I had thought the derivative would be zero for the true parameter value. Thanks for the explanation, appreciate it! :-) | |
| Jun 30, 2023 at 16:56 | comment | added | Iosif Pinelis | @JNL : Is the matter now clear to you? | |
| Jun 29, 2023 at 21:57 | comment | added | Iosif Pinelis | Previous comment continued: But the root $\hat\theta_m$ of $L'$ is -- not the constant $\theta_0$ -- but a random variable, whose values depend on the sample values $z_1,\dots,z_m$. The MLE $\hat\theta_m$ will almost never be equal to the true value $\theta_0$ of the parameter. Instead, as in the particular setting considered in my answer, the MLE $\hat\theta_m$ will, under suitable conditions converge to the true value $\theta_0$ of the parameter in probability (and even almost surely) as $m\to\infty$. | |
| Jun 29, 2023 at 21:57 | comment | added | Iosif Pinelis | @JNL : Of course, the derivative $L'(\theta_0)$ of the log-likelihood function $L$ at the true value $\theta_0$ of the parameter will almost never be zero. That derivative, $L'(\theta_0)$, is a random variable whose values depend on the sample values $z_1,\dots,z_m$, and in problems like this $L'(\theta_0)$ will take the zero value only with probability $0$. On the other hand, you will have $L'(\hat\theta_m)=0$, where $\hat\theta_m$ is the maximum likelihood estimator (MLE). | |
| Jun 29, 2023 at 18:05 | comment | added | JNL | Sorry, I mean "at the true parameter value". If a particular parameter value $\theta=\theta_0$ produces a sample of the random variable $z$, then we ought to be able to estimate $\theta_0$ by maximizing the LL function (i.e. by finding the zero derivative point), the MLE as you mentioned. So, now if $\sigma_j^{2}=e^{\theta}$ is a valid covariance function (let's say for a distribution of only positive $\theta$), then the derivative of the LL won't be zero at $\theta_0$, I think? | |
| Jun 29, 2023 at 17:42 | comment | added | Iosif Pinelis | What do you mean by "at the parameter"? | |
| Jun 29, 2023 at 17:18 | comment | added | JNL | if I were to choose $\sigma_j^{2}=e^{\theta}$ (I think this would this be a valid function for the covariance as it would always be positive for all positive $\theta$?) then the LL derivative would be -0.5 at the parameter (I think?). Maybe there is something completely wrong in my understanding of this? | |
| Jun 29, 2023 at 16:46 | comment | added | JNL | Sorry, I don't think it has cleared the confusion I have. I can see how the LL function derivative will be zero in your answer. But that considers $\sigma_i^{2}=\theta$. What I am wondering is: in general, given the rearrangement of the LL function derivative in my previous comment, would the the LL function derivative be zero at the parameter for any valid choice of $\sigma_i^{2}$ function? | |
| Jun 29, 2023 at 16:40 | history | edited | Iosif Pinelis | CC BY-SA 4.0 |
deleted 28 characters in body
|
| Jun 29, 2023 at 16:40 | comment | added | Iosif Pinelis | @JNL : Does my answer clear your confusion? If not, why not, specifically? | |
| Jun 29, 2023 at 16:36 | comment | added | JNL | By the "first term", I mean $-\frac{1}{2}(\sigma_j^{2}(\theta))^{-1}\frac{\partial \sigma_j^{2}(\theta)}{\partial \theta}$ in the summation. It is confusing (or unclear) to me because if I rewrite the log-likelihood function as: $\frac{\partial L}{\partial \theta}= \sum_{j=1}^{m}-\frac{1}{2\sigma_j^2(\theta)} \frac{\partial \sigma_j^2}{\partial \theta}\left[ 1 + \frac{(z_j-\mu_j(\theta))^2}{\sigma_j^2(\theta)} \right]-\frac{(z_j-\mu_j(\theta))}{\sigma_j^2(\theta)}\frac{\partial \mu_j}{\partial \theta}$ it is not immediately clear to me that it would be zero for any covariance function. | |
| Jun 29, 2023 at 14:38 | history | answered | Iosif Pinelis | CC BY-SA 4.0 |