Derivative of log-likelihood function for Gaussian distribution with parameterized variance

Suppose we have a parameter $$\theta \in R^{n}$$ that defines some noisy observation $$z=\mu(\theta)+\eta, z\in R^{m}$$ where the noise follows a Gaussian distribution whose covariance is a function of the parameter, i.e. $$\eta\sim N(0,\Sigma(\theta))$$. Then the observation random variable is $$z \sim N(\mu(\theta),\Sigma(\theta))$$. Assume observations of $$z$$ are iid and $$\Sigma=diag(\sigma_1^{2}(\theta), \sigma_2^{2}(\theta)...\sigma_m^{2}(\theta))$$, where $$\sigma_i^{2}\ne\sigma_j^{2}$$.

Then, the log-likelihood function would be:

\begin{align} L_z\left(z;\theta\right)&=log\left(\prod^m_{j=1}\frac{1}{\sqrt{2 \pi \sigma_j^2\left(\theta\right)}} e^{-\frac{\left(z_j-\mu_j(\theta)\right)^2}{2 \sigma_j^2\left(\theta\right)}}\right)\\ &=\sum_{j=1}^{m} -\frac{1}{2}log(2\pi)-\frac{1}{2} log(\sigma_{j}^{2}(\theta))-\frac{(z_j-\mu_j(\theta))^{2}}{2\sigma_j^{2}(\theta)}\\ \end{align}

The first derivative of the log-likelihood function w.r.t the parameters would be:

\begin{aligned} \frac{\partial L}{\partial \theta}= \sum_{j=1}^{m}-\frac{1}{2}\left(\sigma_j^2(\theta)\right)^{-1} \frac{\partial \sigma_j^2}{\partial \theta}-(z_j-\mu_j(\theta)) \frac{\partial \mu_j}{\partial \theta}\left(\sigma_j^2(\theta)\right)^{-1}-\frac{1}{2}\left(\sigma_j^2(\theta)\right)^{-2} (z_j-\mu_j(\theta))^2 \frac{\partial \sigma_j^2(\theta)}{\partial \theta} \end{aligned}

What I'm getting confused about is: how would the log-likelihood function derivative be zero at the parameter value, when the first term of the derivative does not depend on $$(z_j-\mu_j(\theta))$$?

$$\newcommand\th\theta\newcommand\si\sigma\newcommand\p\partial\newcommand\ol\overline$$There is no reason to get confused here. Indeed, that "the first term of the derivative does not depend on $$(z_j-\mu_j(\theta))$$" does not at all prevent the derivative from taking the zero value.

If e.g. $$\mu(\th)=\th$$ and $$\si_i(\th)=\th$$ for all real $$\th>0$$ and all $$i=1,\dots,m$$, then $$\frac{\p L}{\p\th}=-\frac n{\th^3}\,(-\ol{z^2}+\th\ol z+\th^2),$$ where $$\ol z:=\frac1m\,\sum_{i=1}^m z_i$$ and $$\ol{z^2}:=\frac1m\,\sum_{i=1}^m z_i^2$$, so that $$\frac{\p L}{\p\th}=0$$ at a real $$\th>0$$ if and only if $$\th=\hat\th:=\hat\th_m:=\frac{-\ol z+\sqrt{\ol z^2+4\ol{z^2}}}2.$$

Moreover, here $$\hat\th$$ is the maximum likelihood estimator of $$\th$$. Using (say) the law of large numbers, one can easily check that $$\hat\th_m$$ is consistent: $$\hat\th_m\to\th$$ in probability as $$m\to\infty$$ (assuming that $$\th$$ is the true value of the parameter).

• By the "first term", I mean $-\frac{1}{2}(\sigma_j^{2}(\theta))^{-1}\frac{\partial \sigma_j^{2}(\theta)}{\partial \theta}$ in the summation. It is confusing (or unclear) to me because if I rewrite the log-likelihood function as: $\frac{\partial L}{\partial \theta}= \sum_{j=1}^{m}-\frac{1}{2\sigma_j^2(\theta)} \frac{\partial \sigma_j^2}{\partial \theta}\left[ 1 + \frac{(z_j-\mu_j(\theta))^2}{\sigma_j^2(\theta)} \right]-\frac{(z_j-\mu_j(\theta))}{\sigma_j^2(\theta)}\frac{\partial \mu_j}{\partial \theta}$ it is not immediately clear to me that it would be zero for any covariance function.
– JNL
Commented Jun 29, 2023 at 16:36
• @JNL : Does my answer clear your confusion? If not, why not, specifically? Commented Jun 29, 2023 at 16:40
• Sorry, I don't think it has cleared the confusion I have. I can see how the LL function derivative will be zero in your answer. But that considers $\sigma_i^{2}=\theta$. What I am wondering is: in general, given the rearrangement of the LL function derivative in my previous comment, would the the LL function derivative be zero at the parameter for any valid choice of $\sigma_i^{2}$ function?
– JNL
Commented Jun 29, 2023 at 16:46
• @JNL : Of course, the derivative $L'(\theta_0)$ of the log-likelihood function $L$ at the true value $\theta_0$ of the parameter will almost never be zero. That derivative, $L'(\theta_0)$, is a random variable whose values depend on the sample values $z_1,\dots,z_m$, and in problems like this $L'(\theta_0)$ will take the zero value only with probability $0$. On the other hand, you will have $L'(\hat\theta_m)=0$, where $\hat\theta_m$ is the maximum likelihood estimator (MLE). Commented Jun 29, 2023 at 21:57
• Previous comment continued: But the root $\hat\theta_m$ of $L'$ is -- not the constant $\theta_0$ -- but a random variable, whose values depend on the sample values $z_1,\dots,z_m$. The MLE $\hat\theta_m$ will almost never be equal to the true value $\theta_0$ of the parameter. Instead, as in the particular setting considered in my answer, the MLE $\hat\theta_m$ will, under suitable conditions converge to the true value $\theta_0$ of the parameter in probability (and even almost surely) as $m\to\infty$. Commented Jun 29, 2023 at 21:57