0
$\begingroup$

Suppose we have a parameter $\theta \in R^{n}$ that defines some noisy observation $z=\mu(\theta)+\eta, z\in R^{m}$ where the noise follows a Gaussian distribution whose covariance is a function of the parameter, i.e. $\eta\sim N(0,\Sigma(\theta))$. Then the observation random variable is $z \sim N(\mu(\theta),\Sigma(\theta))$. Assume observations of $z$ are iid and $\Sigma=diag(\sigma_1^{2}(\theta), \sigma_2^{2}(\theta)...\sigma_m^{2}(\theta))$, where $\sigma_i^{2}\ne\sigma_j^{2}$.

Then, the log-likelihood function would be:

\begin{align} L_z\left(z;\theta\right)&=log\left(\prod^m_{j=1}\frac{1}{\sqrt{2 \pi \sigma_j^2\left(\theta\right)}} e^{-\frac{\left(z_j-\mu_j(\theta)\right)^2}{2 \sigma_j^2\left(\theta\right)}}\right)\\ &=\sum_{j=1}^{m} -\frac{1}{2}log(2\pi)-\frac{1}{2} log(\sigma_{j}^{2}(\theta))-\frac{(z_j-\mu_j(\theta))^{2}}{2\sigma_j^{2}(\theta)}\\ \end{align}

The first derivative of the log-likelihood function w.r.t the parameters would be:

\begin{aligned} \frac{\partial L}{\partial \theta}= \sum_{j=1}^{m}-\frac{1}{2}\left(\sigma_j^2(\theta)\right)^{-1} \frac{\partial \sigma_j^2}{\partial \theta}-(z_j-\mu_j(\theta)) \frac{\partial \mu_j}{\partial \theta}\left(\sigma_j^2(\theta)\right)^{-1}-\frac{1}{2}\left(\sigma_j^2(\theta)\right)^{-2} (z_j-\mu_j(\theta))^2 \frac{\partial \sigma_j^2(\theta)}{\partial \theta} \end{aligned}

What I'm getting confused about is: how would the log-likelihood function derivative be zero at the parameter value, when the first term of the derivative does not depend on $(z_j-\mu_j(\theta))$?

$\endgroup$

1 Answer 1

2
$\begingroup$

$\newcommand\th\theta\newcommand\si\sigma\newcommand\p\partial\newcommand\ol\overline$There is no reason to get confused here. Indeed, that "the first term of the derivative does not depend on $(z_j-\mu_j(\theta))$" does not at all prevent the derivative from taking the zero value.

If e.g. $\mu(\th)=\th$ and $\si_i(\th)=\th$ for all real $\th>0$ and all $i=1,\dots,m$, then $$\frac{\p L}{\p\th}=-\frac n{\th^3}\,(-\ol{z^2}+\th\ol z+\th^2),$$ where $\ol z:=\frac1m\,\sum_{i=1}^m z_i$ and $\ol{z^2}:=\frac1m\,\sum_{i=1}^m z_i^2$, so that $\frac{\p L}{\p\th}=0$ at a real $\th>0$ if and only if $$\th=\hat\th:=\hat\th_m:=\frac{-\ol z+\sqrt{\ol z^2+4\ol{z^2}}}2.$$

Moreover, here $\hat\th$ is the maximum likelihood estimator of $\th$. Using (say) the law of large numbers, one can easily check that $\hat\th_m$ is consistent: $\hat\th_m\to\th$ in probability as $m\to\infty$ (assuming that $\th$ is the true value of the parameter).

$\endgroup$
11
  • $\begingroup$ By the "first term", I mean $-\frac{1}{2}(\sigma_j^{2}(\theta))^{-1}\frac{\partial \sigma_j^{2}(\theta)}{\partial \theta}$ in the summation. It is confusing (or unclear) to me because if I rewrite the log-likelihood function as: $\frac{\partial L}{\partial \theta}= \sum_{j=1}^{m}-\frac{1}{2\sigma_j^2(\theta)} \frac{\partial \sigma_j^2}{\partial \theta}\left[ 1 + \frac{(z_j-\mu_j(\theta))^2}{\sigma_j^2(\theta)} \right]-\frac{(z_j-\mu_j(\theta))}{\sigma_j^2(\theta)}\frac{\partial \mu_j}{\partial \theta}$ it is not immediately clear to me that it would be zero for any covariance function. $\endgroup$
    – JNL
    Commented Jun 29, 2023 at 16:36
  • $\begingroup$ @JNL : Does my answer clear your confusion? If not, why not, specifically? $\endgroup$ Commented Jun 29, 2023 at 16:40
  • $\begingroup$ Sorry, I don't think it has cleared the confusion I have. I can see how the LL function derivative will be zero in your answer. But that considers $\sigma_i^{2}=\theta$. What I am wondering is: in general, given the rearrangement of the LL function derivative in my previous comment, would the the LL function derivative be zero at the parameter for any valid choice of $\sigma_i^{2}$ function? $\endgroup$
    – JNL
    Commented Jun 29, 2023 at 16:46
  • 1
    $\begingroup$ @JNL : Of course, the derivative $L'(\theta_0)$ of the log-likelihood function $L$ at the true value $\theta_0$ of the parameter will almost never be zero. That derivative, $L'(\theta_0)$, is a random variable whose values depend on the sample values $z_1,\dots,z_m$, and in problems like this $L'(\theta_0)$ will take the zero value only with probability $0$. On the other hand, you will have $L'(\hat\theta_m)=0$, where $\hat\theta_m$ is the maximum likelihood estimator (MLE). $\endgroup$ Commented Jun 29, 2023 at 21:57
  • 1
    $\begingroup$ Previous comment continued: But the root $\hat\theta_m$ of $L'$ is -- not the constant $\theta_0$ -- but a random variable, whose values depend on the sample values $z_1,\dots,z_m$. The MLE $\hat\theta_m$ will almost never be equal to the true value $\theta_0$ of the parameter. Instead, as in the particular setting considered in my answer, the MLE $\hat\theta_m$ will, under suitable conditions converge to the true value $\theta_0$ of the parameter in probability (and even almost surely) as $m\to\infty$. $\endgroup$ Commented Jun 29, 2023 at 21:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.