0
$\begingroup$

Suppose we have a parameter $\theta \in R^{n}$ that defines some noisy observation $z=\mu(\theta)+\eta, z\in R^{m}$ where the noise follows a Gaussian distribution whose covariance is a function of the parameter, i.e. $\eta\sim N(0,\Sigma(\theta))$. Then the observation random variable is $z \sim N(\mu(\theta),\Sigma(\theta))$. Assume observations of $z$ are iid and $\Sigma=diag(\sigma_1^{2}(\theta), \sigma_2^{2}(\theta)...\sigma_m^{2}(\theta))$, where $\sigma_i^{2}\ne\sigma_j^{2}$.

Then, the log-likelihood function would be:

\begin{align} L_z\left(z;\theta\right)&=log\left(\prod^m_{j=1}\frac{1}{\sqrt{2 \pi \sigma_j^2\left(\theta\right)}} e^{-\frac{\left(z_j-\mu_j(\theta)\right)^2}{2 \sigma_j^2\left(\theta\right)}}\right)\\ &=\sum_{j=1}^{m} -\frac{1}{2}log(2\pi)-\frac{1}{2} log(\sigma_{j}^{2}(\theta))-\frac{(z_j-\mu_j(\theta))^{2}}{2\sigma_j^{2}(\theta)}\\ \end{align}

The first derivative of the log-likelihood function w.r.t the parameters would be:

\begin{aligned} \frac{\partial L}{\partial \theta}= \sum_{j=1}^{m}-\frac{1}{2}\left(\sigma_j^2(\theta)\right)^{-1} \frac{\partial \sigma_j^2}{\partial \theta}-(z_j-\mu_j(\theta)) \frac{\partial \mu_j}{\partial \theta}\left(\sigma_j^2(\theta)\right)^{-1}-\frac{1}{2}\left(\sigma_j^2(\theta)\right)^{-2} (z_j-\mu_j(\theta))^2 \frac{\partial \sigma_j^2(\theta)}{\partial \theta} \end{aligned}

What I'm getting confused about is: how would the log-likelihood function derivative be zero at the parameter value, when the first term of the derivative does not depend on $(z_j-\mu_j(\theta))$?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

$\newcommand\th\theta\newcommand\si\sigma\newcommand\p\partial\newcommand\ol\overline$There is no reason to get confused here. Indeed, that "the first term of the derivative does not depend on $(z_j-\mu_j(\theta))$" does not at all prevent the derivative from taking the zero value.

If e.g. $\mu(\th)=\th$ and $\si_i(\th)=\th$ for all real $\th>0$ and all $i=1,\dots,m$, then $$\frac{\p L}{\p\th}=-\frac n{\th^3}\,(-\ol{z^2}+\th\ol z+\th^2),$$ where $\ol z:=\frac1m\,\sum_{i=1}^m z_i$ and $\ol{z^2}:=\frac1m\,\sum_{i=1}^m z_i^2$, so that $\frac{\p L}{\p\th}=0$ at a real $\th>0$ if and only if $$\th=\hat\th:=\hat\th_m:=\frac{-\ol z+\sqrt{\ol z^2+4\ol{z^2}}}2.$$

Moreover, here $\hat\th$ is the maximum likelihood estimator of $\th$. Using (say) the law of large numbers, one can easily check that $\hat\th_m$ is consistent: $\hat\th_m\to\th$ in probability as $m\to\infty$ (assuming that $\th$ is the true value of the parameter).

Improve this answer
$\endgroup$
11
  • $\begingroup$ By the "first term", I mean $-\frac{1}{2}(\sigma_j^{2}(\theta))^{-1}\frac{\partial \sigma_j^{2}(\theta)}{\partial \theta}$ in the summation. It is confusing (or unclear) to me because if I rewrite the log-likelihood function as: $\frac{\partial L}{\partial \theta}= \sum_{j=1}^{m}-\frac{1}{2\sigma_j^2(\theta)} \frac{\partial \sigma_j^2}{\partial \theta}\left[ 1 + \frac{(z_j-\mu_j(\theta))^2}{\sigma_j^2(\theta)} \right]-\frac{(z_j-\mu_j(\theta))}{\sigma_j^2(\theta)}\frac{\partial \mu_j}{\partial \theta}$ it is not immediately clear to me that it would be zero for any covariance function. $\endgroup$
    – JNL
    Commented Jun 29, 2023 at 16:36
  • $\begingroup$ @JNL : Does my answer clear your confusion? If not, why not, specifically? $\endgroup$
    – Iosif Pinelis
    Commented Jun 29, 2023 at 16:40
  • $\begingroup$ Sorry, I don't think it has cleared the confusion I have. I can see how the LL function derivative will be zero in your answer. But that considers $\sigma_i^{2}=\theta$. What I am wondering is: in general, given the rearrangement of the LL function derivative in my previous comment, would the the LL function derivative be zero at the parameter for any valid choice of $\sigma_i^{2}$ function? $\endgroup$
    – JNL
    Commented Jun 29, 2023 at 16:46
  • 1
    $\begingroup$ @JNL : Of course, the derivative $L'(\theta_0)$ of the log-likelihood function $L$ at the true value $\theta_0$ of the parameter will almost never be zero. That derivative, $L'(\theta_0)$, is a random variable whose values depend on the sample values $z_1,\dots,z_m$, and in problems like this $L'(\theta_0)$ will take the zero value only with probability $0$. On the other hand, you will have $L'(\hat\theta_m)=0$, where $\hat\theta_m$ is the maximum likelihood estimator (MLE). $\endgroup$
    – Iosif Pinelis
    Commented Jun 29, 2023 at 21:57
  • 1
    $\begingroup$ Previous comment continued: But the root $\hat\theta_m$ of $L'$ is -- not the constant $\theta_0$ -- but a random variable, whose values depend on the sample values $z_1,\dots,z_m$. The MLE $\hat\theta_m$ will almost never be equal to the true value $\theta_0$ of the parameter. Instead, as in the particular setting considered in my answer, the MLE $\hat\theta_m$ will, under suitable conditions converge to the true value $\theta_0$ of the parameter in probability (and even almost surely) as $m\to\infty$. $\endgroup$
    – Iosif Pinelis
    Commented Jun 29, 2023 at 21:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .