Skip to main content
added 31 characters in body
Source Link
Michael Hardy
  • 1
  • 12
  • 85
  • 126

A ``down-to-earth'' observation to see what goes wrong with method of moments is this:

When considering applying the method of moment to $(X_1,...,X_n)$, you may as well apply the method of moments to transformed data-points $(g(X_1),...,g(X_n))$ for any, say, smooth bijection $g$. So which $g$ is the right choice? It is easy to see on particular examples (e.g., $X_i\sim N(\theta,1)$) that choosing a wrong $g$ can make the asymptotic variance much worse.

The answer to ``what is the right function $g$'' leads to the theory of minimal sufficient statistics and of the MLE, that provide estimators that do not depend on a particular choice of transformation $g$. For instance if $X_i\sim f_\theta$ for some density $f_\theta$, the log-likelihood for $(X_1,...,X_n)$$(X_1,\ldots,X_n)$ is $\theta\mapsto\sum_i \log(f_\theta(X_i))$ and the log-likelihood for $(Y_1,...,Y_n)=(g(X_1),...,g(X_n))$$(Y_1,\ldots,Y_n)=(g(X_1),\ldots,g(X_n))$ (which have density $F_\theta(y)=f_\theta(g^{-1}(y)) |\det(\nabla g^{-1}(y))|$$F_\theta(y)=f_\theta(g^{-1}(y)) \left|\det(\nabla g^{-1}(y))\right|$ is $$ \theta\mapsto \sum_i \log F_\theta(Y_i) = \sum_i \log f_\theta(X_i) + \sum_i \log |\det(\nabla g^{-1}(Y_i))|. $$$$ \theta\mapsto \sum_i \log F_\theta(Y_i) = \sum_i \log f_\theta(X_i) + \sum_i \log \left|\det(\nabla g^{-1}(Y_i))\right|. $$ The second term is constant so maximizing with respect to $\theta$ either log-likelihood gives the same $\hat\theta$.

A ``down-to-earth'' observation to see what goes wrong with method of moments is this:

When considering applying the method of moment to $(X_1,...,X_n)$, you may as well apply the method of moments to transformed data-points $(g(X_1),...,g(X_n))$ for any, say, smooth bijection $g$. So which $g$ is the right choice? It is easy to see on particular examples (e.g., $X_i\sim N(\theta,1)$) that choosing a wrong $g$ can make the asymptotic variance much worse.

The answer to ``what is the right function $g$'' leads to the theory of minimal sufficient statistics and of the MLE, that provide estimators that do not depend on a particular choice of transformation $g$. For instance if $X_i\sim f_\theta$ for some density $f_\theta$, the log-likelihood for $(X_1,...,X_n)$ is $\theta\mapsto\sum_i \log(f_\theta(X_i))$ and the log-likelihood for $(Y_1,...,Y_n)=(g(X_1),...,g(X_n))$ (which have density $F_\theta(y)=f_\theta(g^{-1}(y)) |\det(\nabla g^{-1}(y))|$ is $$ \theta\mapsto \sum_i \log F_\theta(Y_i) = \sum_i \log f_\theta(X_i) + \sum_i \log |\det(\nabla g^{-1}(Y_i))|. $$ The second term is constant so maximizing with respect to $\theta$ either log-likelihood gives the same $\hat\theta$.

A ``down-to-earth'' observation to see what goes wrong with method of moments is this:

When considering applying the method of moment to $(X_1,...,X_n)$, you may as well apply the method of moments to transformed data-points $(g(X_1),...,g(X_n))$ for any, say, smooth bijection $g$. So which $g$ is the right choice? It is easy to see on particular examples (e.g., $X_i\sim N(\theta,1)$) that choosing a wrong $g$ can make the asymptotic variance much worse.

The answer to ``what is the right function $g$'' leads to the theory of minimal sufficient statistics and of the MLE, that provide estimators that do not depend on a particular choice of transformation $g$. For instance if $X_i\sim f_\theta$ for some density $f_\theta$, the log-likelihood for $(X_1,\ldots,X_n)$ is $\theta\mapsto\sum_i \log(f_\theta(X_i))$ and the log-likelihood for $(Y_1,\ldots,Y_n)=(g(X_1),\ldots,g(X_n))$ (which have density $F_\theta(y)=f_\theta(g^{-1}(y)) \left|\det(\nabla g^{-1}(y))\right|$ is $$ \theta\mapsto \sum_i \log F_\theta(Y_i) = \sum_i \log f_\theta(X_i) + \sum_i \log \left|\det(\nabla g^{-1}(Y_i))\right|. $$ The second term is constant so maximizing with respect to $\theta$ either log-likelihood gives the same $\hat\theta$.

Source Link
jlewk
  • 1.7k
  • 7
  • 14

A ``down-to-earth'' observation to see what goes wrong with method of moments is this:

When considering applying the method of moment to $(X_1,...,X_n)$, you may as well apply the method of moments to transformed data-points $(g(X_1),...,g(X_n))$ for any, say, smooth bijection $g$. So which $g$ is the right choice? It is easy to see on particular examples (e.g., $X_i\sim N(\theta,1)$) that choosing a wrong $g$ can make the asymptotic variance much worse.

The answer to ``what is the right function $g$'' leads to the theory of minimal sufficient statistics and of the MLE, that provide estimators that do not depend on a particular choice of transformation $g$. For instance if $X_i\sim f_\theta$ for some density $f_\theta$, the log-likelihood for $(X_1,...,X_n)$ is $\theta\mapsto\sum_i \log(f_\theta(X_i))$ and the log-likelihood for $(Y_1,...,Y_n)=(g(X_1),...,g(X_n))$ (which have density $F_\theta(y)=f_\theta(g^{-1}(y)) |\det(\nabla g^{-1}(y))|$ is $$ \theta\mapsto \sum_i \log F_\theta(Y_i) = \sum_i \log f_\theta(X_i) + \sum_i \log |\det(\nabla g^{-1}(Y_i))|. $$ The second term is constant so maximizing with respect to $\theta$ either log-likelihood gives the same $\hat\theta$.