0
$\begingroup$

My question concerns using Maximum Likelihood to estimate unknown parameters. I will sincerely appreciate if anyone can help me find out if my approach is flawed.

Assume we have a random vector $V$, and we can observe $M$ samples of it denoted by $V_1,V_2,\ldots,V_M$. Define a scalar random variable $X = f(V\mid\theta)$ where $f(\cdot)$ is function whose closed form is known but parameter $\theta$ is unknown. I cannot observe $X$.

We do not know the distribution of $V$, but we do know the distribution of $X$. In fact, $X$ follows a Normal distribution $N(\mu,\sigma^2)$ where $\mu$ and $\sigma$ are unknown.

My purpose is to JOINTLY estimate $\theta$, $\mu$, and $\sigma$. I take the following approach: let us use $g(.|\mu,\sigma^2)$ to denote the pdf for $N(\mu,\sigma)$. The likelihood function $L$ is then $g(f(V\mid\theta)\mid\mu,\sigma^2)$. Based on MLE, we then maximize $\sum_{m=1}^M \log(g(f(V_m\mid\theta)\mid\mu,\sigma^2))$ of all $M$ samples. Since $f(\cdot)$ and $g(\cdot)$ are both known, this maximization problem can be solved by nonlinear programming (at least in theory). The optimal solution would provide the best estimates for $\theta$, $\mu$, and $\sigma$ simultaneously.

I have multiple questions regarding the approach described above:

  1. Deviation from traditional MLE definition: in textbook examples, we should formulate the joint distribution of $V$ (joint because $V$ is a vector) and maximize it. In my problem, however, the joint distribution of $V$ is unknown. The only thing we know is that it has to satisfy $X = f(V)$. For example, let us assume $V$ has two dimensions, and $ X = V_{(1)} + \theta V_{(2)}$. There are many joint distributions that satisfy this equation above.

  2. Identification: as pointed by the example above, There is no one-to-one mapping between $V$ and $X$. Do I have to run into identification issues when I do the estimation?

  3. Do consistency, asymptotic normality and other decent properties for MLE still apply?

  4. Is there any alternative besides MLE I can take?

Many thanks! Any comments are more than welcome.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

The correct usage is "a sample of size $M$", rather than "$M$ samples".

The function $f$ and the probability distribution of $V$ would completely determine the distribution of $X = f(V\mid\theta)$, and therefore would completely determine $\mu$ and $\sigma$. Hence form some function $h$ we have $h(\theta) = (\mu,\sigma)$. Since the parameter $\theta$ then determines the probability distribution of $X$, if there is such a thing as the MLE $\hat{\theta}$ for $\theta$, then $h(\hat{\theta})$ would be the MLE for the pair $(\mu,\sigma)$. So you can't treat $\theta$ and the pair $(\mu,\sigma)$ as three independent parameters to be estimated.

Your assertion about what the likelihood function is doesn't make sense. You could say $L_1(\mu,\sigma) = g(X\mid\mu,\sigma)$ is a likelihood function. How to find the values of $\mu$ and $\sigma$ that maximize that is well-known. But in order to find a likelihood function $L_2(\theta)$ you would need to know a way in which the probability distribution of the data depends on $\theta$. You seem to be saying you don't know that.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for the comments. My notations are to be improved, but hopefully you can understand what I mean. The gist of your point seems to be that, if I know the distribution of $V$, the distribution of $X$ can be derived. If it is indeed your point I agree with you. But in my case, the distribution of $V$ is not known. I guess I would rephase my question in the following way: if we can observe samples of $V$ but its closed-form distribution is unknown, and the distribution of $f(V|\theta)$ is known (except for $\theta$), what kind of parameter estimates can we do for $\theta$? $\endgroup$
    – Haining Yu
    Mar 31, 2011 at 21:54
  • $\begingroup$ I'm not altogether sure I do understand what you mean. I think you said (among other things) that $X = f(V \mid \theta)$ does not depend in a one-to-one fashion upon $V$. If $f$ is known, and if the dependence of $X$ on $V$ were one-to-one, then the probability distribution of $V$ would essentially be known if $\mu$ and $\sigma$ were known, in the sense that there would be only one probability distribution of $V$ that would give rise to that particular probability distribution of $X$. The parameter $\theta$ would then simply be a function of $(\mu,\sigma)$, at least if...... $\endgroup$
    – Michael Hardy
    Apr 1, 2011 at 0:02
  • $\begingroup$ ....two different $\theta$'s never correspond to the same distribution of $V$. If $\theta$ is a function of $(\mu,\sigma)$---say $\theta = j(\mu,\sigma)$, then the MLE for $\theta$ would just be $j(\hat{\mu},\hat{\sigma})$, where $\hat{\mu}$ and $\hat{\sigma}$ are the respective MLEs of $\mu$ and $\simga$. So the difficulty that still remains comes only from the non-one-to-one nature of the dependence of $X$ on $V$. $\endgroup$
    – Michael Hardy
    Apr 1, 2011 at 0:05
  • $\begingroup$ Sorry about your confusion. Please let me clarify: 1. there exists a well-defined function $f(.)$ so that $x=f(V|\theta)$. Therefore if the pdf of V is known, the pdf of x is known. 2. The pdf of $V$ cannot be uniquely determined by the pdf of $x$ and function $f(.)$. 3. $x=f(V|\theta)$ is set up that, given enough pairs of x and V, $\theta$ can be uniquely identified Does that help? $\endgroup$
    – Haining Yu
    Apr 1, 2011 at 0:41
  • $\begingroup$ I don't think you've got a well-defined math problem here. $V$ could be a vector of 100 independent normally distributed random variables with uniform distributions on the interval $(0,1)$ and $X$ could be the sum of $\theta\in\mathbb{R}$ and a suitable function of the 43rd one of those uniform RVs, so that you get a normally distributed RV with expectation $\theta$. Or you could do all sorts of things like that. At best, you'd have to say more than what you've said to specify a mathematical problem. $\endgroup$
    – Michael Hardy
    Apr 1, 2011 at 22:00
1
$\begingroup$

You definitely would run into identification issues. Suppose $f(x|\mid\theta) = x \theta$. Then multiplying $\theta$ and $\sigma$ by any common factor would result in an identical situation. In this case the MLE would not really be defined.

Generally, as $\theta$ changes, $X$ is revealing a different facet of $V$. For example, if $f$ is a projection map and $\theta$ determines the angle of projection, then $f(V\mid\theta)$ is the shadow of $V$ in the direction of $\theta$. Many of the shadows could be normally distributed, and could be essentially unrelated to each other.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the comment. I see there are possibility for multiple identification issues. First on the scaling issue: in my real problem I have ways to prevent $\mu$ and $\sigma$ from cancelling each other. In fact I can set $\mu$ to a constant. I am indeed more concerned by the relationship between $X$ and $V$. For example, assume $ X = sum_{k}{v_k\theta_k} + c $ where $c$ is a non-zero constant. When I have a large (or infinite) number of samples, $\theta$ should be unique identified? $\endgroup$
    – Haining Yu
    Mar 31, 2011 at 20:48
  • $\begingroup$ Suppose that the V are normally distributed (not necessarily independent). Then for any choice of $\theta_k$, the resulting $X$ is normally distributed with mean and variance depending on $\theta$. There is no such thing as the "correct" $\theta$. $\endgroup$
    – David Harris
    Apr 1, 2011 at 13:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.