4
$\begingroup$

Under appropriate regularity conditions it is well-known that Maximum Likelihood Estimation (MLE) produces asymptotically efficient estimators in the sense that their asymptotic covariance is given by the inverse of Fisher information, i.e. $$ \sqrt n(\theta-\tilde\theta)\overset{d}{\to}\mathcal N(0,I^{-1}(\theta)) $$ as $n\to\infty$.

With the Method of Moments (MoM) this is not necessarily true. There are cases where MLE and MoM produce the same estimators; however, in general, there is no guarantee of asymptotic efficiency in the MoM's.

My question is why? Intuitively, MLE requires specifying the exact distribution from which the observed data is realized. On the other hand, MoM only requires specifying the first $m$-moments ($\theta\in\Bbb R^m$) of the data's distribution. So I suspect this lack of specificity is why we do not have any guarantees of asymptotic efficiency. Is this intuition correct? Can someone spell this out with a more compelling theoretical argument?

$\endgroup$
23
  • 3
    $\begingroup$ Moments are not necessarily sufficient statistics, while the likelihood function always is sufficient? For instance, method of moments for the "German tank problem" mathoverflow.net/questions/14964/… leads to estimation based on the mean, which is not sufficient in this case. $\endgroup$ Commented Aug 1, 2023 at 17:50
  • 1
    $\begingroup$ @kjetilbhalvorsen: The likelihood function is not even a statistic. A technically correct version of what I think you're saying is that if there is a sufficient statistic, a unique MLE will be a function of it. $\endgroup$ Commented Aug 1, 2023 at 21:32
  • 1
    $\begingroup$ @MichaelHardy: A statistic is, by definition, a random variable that is a function of the random sample. The likelihood function depends on the parameter. (If it doesn't, then any statistic is sufficient by Neyman's criterion.) $\endgroup$ Commented Aug 2, 2023 at 0:41
  • 1
    $\begingroup$ @kjetilbhalvorsen Your original comment seems to be the prevailing reason for the performance differences between the two estimation methods. The data reduction portion of MoM's estimation (raw sample$\to$ moments) is generally not sufficient and incurs information loss. On the other hand, the data reduction of MLE (raw sample$\to$likelihood function) induces a minimal sufficient partition of the sample space and thus does not incur a loss of information. $\endgroup$ Commented Nov 8, 2023 at 15:41
  • 1
    $\begingroup$ @MichaelHardy: An obvious example where the mle is not sufficient is iid Cauchy observations (with $n$ larger than 1) $\endgroup$ Commented Nov 8, 2023 at 17:19

2 Answers 2

3
$\begingroup$

The comment by kjetil b halvorsen seems to have a good point. In view of the ensuing discussion, it may be of use to detail, clarify, and complement some of the raised points, which will be done below.

The likelihood function is usually defined as the map $\Theta\ni\theta\mapsto L_x(\theta):= f_\theta(x)$, where $\Theta$ is the parameter space, $x$ is a realization (value) of the random sample $X$ taken from the distribution with density $f_{\theta_0}$, $\theta_0\in\Theta$ is the "true" value of the parameter $\theta$, and $(f_\theta)_{\theta\in\Theta}$ is a family of probability densities (referred to as the statistical model).

So, for each realization $x$ of $X$ we have its own likelihood function $L_x$. We may then consider the function $x\mapsto \mathcal L(x):=L_x$, and we may even want to refer to this function $\mathcal L$ as the likelihood function as well. The function $\mathcal L$ will be measurable with respect to the cylindrical $\sigma$-algebra over $\mathbb R^\Theta$. So, the random function $L_X:=\mathcal L(X):=\mathcal L\circ X$ is a statistic with values in $\mathbb R^\Theta$. Trivially, the statistic $L_X$ is sufficient, since $f_\theta(x)=L_x(\theta)$ for all $\theta$ and $x$. (However, to focus on the essential ideas, let us not be concerned with measurability matters in what follows in this answer.)

So, trivially, the "maximum likelihood estimator (MLE)" $$\operatorname*{argmax}_{\theta\in\Theta}L_X(\theta)$$ with values in the set of all subsets of $\Theta$ is a function of the sufficient statistic $L_X$. This fact is hardly of any significance, though -- because the (entire) sample $X$ is of course always sufficient, and any statistic is, by definition, a function of the sufficient statistic $X$.

What is important is that, by the factorization criterion, the MLE is a function of any sufficient statistic, including minimal sufficient statistics.

(However, the MLE by itself of course does not have to be sufficient. E.g., if $\Theta=(0,\infty)$, $X=(X_1,\dots,X_n)$, $n\ge2$, and $X_1,\dots,X_n$ are i.i.d. normal random variables each with mean $\theta$ and variance $\theta^2$, then the almost surely unique MLE of $\theta$ is \begin{equation} \hat\theta:=\sqrt{\overline X^2/4+\overline{X^2}}-\overline X/2, \end{equation} where $\overline X:=\frac1n\,\sum_1^n X_i$ and $\overline{X^2}:=\frac1n\,\sum_1^n X_i^2$. However, here $(\overline X,\overline{X^2})$ is a minimal sufficient statistic -- which is not a function of $\hat\theta$, and therefore the MLE $\hat\theta$ is not sufficient.)


On the other hand, estimators other than the MLE (including method-of-moment estimators) do not have to be functions of a minimal sufficient statistic, and are therefore "less likely" to have good statistical properties. One way to see this is that, by the Rao–Blackwell theorem, if $S(X)$ is any estimator of $q(\theta)$ for some function $q$ and $T(X)$ is any sufficient statistic, then (i) $S_T(X):=E_\theta(S(X)|T(X))$ is a statistic (as it does not depend on $\theta$); (ii) $S_T(X)$ is a function of the sufficient statistic $T(X)$ (even if $T(X)$ is a minimal sufficient statistic); (iii) the bias of $S_T(X)$ for $q(\theta)$ is the same as the bias of $S(X)$ for $q(\theta)$, for all values of $\theta$; (iv) the variance of $S_T(X)$ is no greater than the variance of $S(X)$, for all values of $\theta$ (and the latter property generalizes to any convex loss function).

So, we can take any estimator which is not a function of a minimal sufficient statistic $T(X)$ and improve it by the described above Rao–Blackwell conditioning on $T(X)$ -- whereas the MLE cannot be improved this way, since the MLE is already a function of any (minimal) sufficient statistic and hence the conditioning on any sufficient statistic does not change the MLE.


Finally, about this:

Intuitively, MLE requires specifying the exact distribution from which the observed data is realized. On the other hand, MoM only requires specifying the first $m$-moments ($\theta\in\Bbb R^m$) of the data's distribution. So I suspect this lack of specificity is why we do not have any guarantees of asymptotic efficiency. Is this intuition correct?

The answer to this is no. Indeed, if $\theta$ is an $m$-dimensional parameter and the method of moments is applicable, then the knowledge of $m$ moments uniquely determines $\theta$, so that you have the complete specificity. E.g., if $f_\theta$ is the density of the gamma distribution with parameter $\theta:=(\alpha,\beta)\in\Theta=(0,\infty)\times(0,\infty)$, then the first two moments $\mu_1(\theta)=\alpha\beta$ and $\mu_2(\theta)=\alpha\beta^2$ uniquely determine $\theta=(\alpha,\beta)$. Another way to look at this is that, in a parametric model, knowing the value of the parameter $\theta$, you fully know the density $f_\theta$ and thus you fully know the corresponding distribution.

$\endgroup$
14
  • $\begingroup$ Is the fact that the MLE is always a function of a minimal sufficient statistic a direct consequence of the likelihood function itself being minimal sufficient? $\endgroup$ Commented Oct 31, 2023 at 17:05
  • $\begingroup$ @AaronHendrickson : In general, the likelihood function is not minimal sufficient. E.g., consider the iid sample of size $n\ge2$ from the Poisson distribution with parameter $\theta$. $\endgroup$ Commented Oct 31, 2023 at 21:52
  • $\begingroup$ @IosifPinelis : I wonder whether the truth value of your latest comment depends on what you consider to be the likelihood function. For the example you mention, I might write $$ L(\lambda) \propto e^{-n\lambda} \lambda^{x_1+\cdots + x_n} $$ and say that's the likelihood function. But I imagine some (you?) might write $\text{“} {=} \text{”}$ rather than $\text{“} {\propto} \text{”}$ and include the reciprocal of the product of factorials, in which case the likelihood function would give some information not relevant to estimating $\lambda. \qquad$ $\endgroup$ Commented Nov 10, 2023 at 14:30
  • $\begingroup$ People who write about this don't always say which of those two conventions they have in mind. $\endgroup$ Commented Nov 10, 2023 at 14:39
  • 1
    $\begingroup$ @MichaelHardy : If the likelihood is defined, as apparently was done in your comment, up to a factor not depending on the parameter, then you do get a minimal sufficient statistic, according to the "useful characterization of minimal sufficiency". $\endgroup$ Commented Nov 10, 2023 at 15:42
3
$\begingroup$

A ``down-to-earth'' observation to see what goes wrong with method of moments is this:

When considering applying the method of moment to $(X_1,...,X_n)$, you may as well apply the method of moments to transformed data-points $(g(X_1),...,g(X_n))$ for any, say, smooth bijection $g$. So which $g$ is the right choice? It is easy to see on particular examples (e.g., $X_i\sim N(\theta,1)$) that choosing a wrong $g$ can make the asymptotic variance much worse.

The answer to ``what is the right function $g$'' leads to the theory of minimal sufficient statistics and of the MLE, that provide estimators that do not depend on a particular choice of transformation $g$. For instance if $X_i\sim f_\theta$ for some density $f_\theta$, the log-likelihood for $(X_1,\ldots,X_n)$ is $\theta\mapsto\sum_i \log(f_\theta(X_i))$ and the log-likelihood for $(Y_1,\ldots,Y_n)=(g(X_1),\ldots,g(X_n))$ (which have density $F_\theta(y)=f_\theta(g^{-1}(y)) \left|\det(\nabla g^{-1}(y))\right|$ is $$ \theta\mapsto \sum_i \log F_\theta(Y_i) = \sum_i \log f_\theta(X_i) + \sum_i \log \left|\det(\nabla g^{-1}(Y_i))\right|. $$ The second term is constant so maximizing with respect to $\theta$ either log-likelihood gives the same $\hat\theta$.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .