Well-definedness of maximum likelihood estimation

Question

Consider a family $\{\mu_\theta:\theta\in\Theta\}$ of probability measures on a measurable space $X$. Given $x\in X$, the maximum likelihood estimate is the value of $\theta$ which maximizes the likelihood of $x$, i.e. which maximizes $\frac{d\mu_\theta}{d\nu}|_x$ for a measure $\nu$ on $X$ with $\mu_\theta\ll\nu$ for all $\theta\in\Theta$.

This is obviously not well-defined since $\frac{d\mu_\theta}{d\nu}$ is only an equivalence class of functions, up to a.e. equality.

In the texts I've seen (e.g. van der Vaart's Asymptotic Statistics) this doesn't seem to be addressed. What is the rigorous measure-theoretic formulation of maximum likelihood estimation, and is there a standard reference where it's explained?

Answer 1

$\newcommand\th\theta\newcommand\R{\mathbb R}$Indeed, the densities $f_\th:=d\mu_\th/d\nu$ are defined only $\nu$-a.e., and hence it makes no sense to define a maximum likelihood estimate (MLE) as a maximizer of $f_\th(x)$ in $\th$ for arbitrary versions of the densities.

Usually, though, it is assumed that versions $f_\th$ of the densities can be chosen so that the likelihood $L_x(\th):=f_\th(x)$ be regular enough in $\th$. In particular, in proofs of the consistency of the MLE, one of the standard assumptions is that the likelihood $L_x(\th)$ be continuous in $\th$; in proofs of central limit theorems for the MLE, one of the standard assumptions is that the likelihood $L_x(\th)$ be twice differentiable in $\th$.

In fact, usually it is a family $(f_\th)$ of densities (regular in $\th$ to some extent) that is assumed, whereas the family $(\mu_\th)$ with $d\mu_\th=f_\th\,d\nu$ may come next, if ever. Also, the choice of the reference measure $\nu$ usually matters little -- because, for any two mutually absolutely continuous reference measures, the likelihood profiles $\th\mapsto L_x(\th)$ will be proportional to each other (with the proportionality coefficient depending only on $x$), which usually will not affect statistical inference based on the likelihood. So, the difficulty that concerns you hardly ever arises.

For some textbook statistical models (that is, for some families $(\mu_\th)$), it is impossible to make $L_x(\th)$ continuous in $\th$. E.g., for real $\th>0$, let $\mu_\th$ be the joint distribution of $n$ iid random variables each with the uniform distribution over the interval $[0,\th]$. Choosing the versions $f_\th$ of the densities of $\mu_\th$ (with respect to the Lebesgue measure over $\R^n$) so that for $x=(x_1,\dots,x_n)\in\R^n$ we have $f_\th(x)=\th^{-n}\,1(\max_i x_i\le\th)$, and so, the MLE is $\max_i x_i$. However, if we choose the versions $f_\th$ of the densities of $\mu_\th$ so that $f_\th(x)=\th^{-n}\,1(\max_i x_i<\th)$, then $L_x(\th)=f_\th(x)$ does not attain a maximum in $\th$ for any $x$, so that now an MLE will not exist, strictly speaking. A generalized definition of an MLE is given in Section 2 of this paper. With that definition, even for $f_\th(x)=\th^{-n}\,1(\max_i x_i<\th)$, $\max_i x_i$ will be the generalized MLE.