0
$\begingroup$

My question concerns using Maximum Likelihood to estimate unknown parameters. I will sincerely appreciate if anyone can help me find out if my approach is flawed.

Assume we have a random vector $V$, and we can observe $M$ samples of it denoted by $V_1,V_2,\ldots,V_M$. Define a scalar random variable $X = f(V\mid\theta)$ where $f(\cdot)$ is function whose closed form is known but parameter $\theta$ is unknown. I cannot observe $X$.

We do not know the distribution of $V$, but we do know the distribution of $X$. In fact, $X$ follows a Normal distribution $N(\mu,\sigma^2)$ where $\mu$ and $\sigma$ are unknown.

My purpose is to JOINTLY estimate $\theta$, $\mu$, and $\sigma$. I take the following approach: let us use $g(.|\mu,\sigma^2)$ to denote the pdf for $N(\mu,\sigma)$. The likelihood function $L$ is then $g(f(V\mid\theta)\mid\mu,\sigma^2)$. Based on MLE, we then maximize $\sum_{m=1}^M \log(g(f(V_m\mid\theta)\mid\mu,\sigma^2))$ of all $M$ samples. Since $f(\cdot)$ and $g(\cdot)$ are both known, this maximization problem can be solved by nonlinear programming (at least in theory). The optimal solution would provide the best estimates for $\theta$, $\mu$, and $\sigma$ simultaneously.

I have multiple questions regarding the approach described above:

  1. Deviation from traditional MLE definition: in textbook examples, we should formulate the joint distribution of $V$ (joint because $V$ is a vector) and maximize it. In my problem, however, the joint distribution of $V$ is unknown. The only thing we know is that it has to satisfy $X = f(V)$. For example, let us assume $V$ has two dimensions, and $ X = V_{(1)} + \theta V_{(2)}$. There are many joint distributions that satisfy this equation above.

  2. Identification: as pointed by the example above, There is no one-to-one mapping between $V$ and $X$. Do I have to run into identification issues when I do the estimation?

  3. Do consistency, asymptotic normality and other decent properties for MLE still apply?

  4. Is there any alternative besides MLE I can take?

Many thanks! Any comments are more than welcome.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

The correct usage is "a sample of size $M$", rather than "$M$ samples".

The function $f$ and the probability distribution of $V$ would completely determine the distribution of $X = f(V\mid\theta)$, and therefore would completely determine $\mu$ and $\sigma$. Hence form some function $h$ we have $h(\theta) = (\mu,\sigma)$. Since the parameter $\theta$ then determines the probability distribution of $X$, if there is such a thing as the MLE $\hat{\theta}$ for $\theta$, then $h(\hat{\theta})$ would be the MLE for the pair $(\mu,\sigma)$. So you can't treat $\theta$ and the pair $(\mu,\sigma)$ as three independent parameters to be estimated.

Your assertion about what the likelihood function is doesn't make sense. You could say $L_1(\mu,\sigma) = g(X\mid\mu,\sigma)$ is a likelihood function. How to find the values of $\mu$ and $\sigma$ that maximize that is well-known. But in order to find a likelihood function $L_2(\theta)$ you would need to know a way in which the probability distribution of the data depends on $\theta$. You seem to be saying you don't know that.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for the comments. My notations are to be improved, but hopefully you can understand what I mean. The gist of your point seems to be that, if I know the distribution of $V$, the distribution of $X$ can be derived. If it is indeed your point I agree with you. But in my case, the distribution of $V$ is not known. I guess I would rephase my question in the following way: if we can observe samples of $V$ but its closed-form distribution is unknown, and the distribution of $f(V|\theta)$ is known (except for $\theta$), what kind of parameter estimates can we do for $\theta$? $\endgroup$
    – Haining Yu
    Commented Mar 31, 2011 at 21:54
  • $\begingroup$ I'm not altogether sure I do understand what you mean. I think you said (among other things) that $X = f(V \mid \theta)$ does not depend in a one-to-one fashion upon $V$. If $f$ is known, and if the dependence of $X$ on $V$ were one-to-one, then the probability distribution of $V$ would essentially be known if $\mu$ and $\sigma$ were known, in the sense that there would be only one probability distribution of $V$ that would give rise to that particular probability distribution of $X$. The parameter $\theta$ would then simply be a function of $(\mu,\sigma)$, at least if...... $\endgroup$
    – Michael Hardy
    Commented Apr 1, 2011 at 0:02
  • $\begingroup$ ....two different $\theta$'s never correspond to the same distribution of $V$. If $\theta$ is a function of $(\mu,\sigma)$---say $\theta = j(\mu,\sigma)$, then the MLE for $\theta$ would just be $j(\hat{\mu},\hat{\sigma})$, where $\hat{\mu}$ and $\hat{\sigma}$ are the respective MLEs of $\mu$ and $\simga$. So the difficulty that still remains comes only from the non-one-to-one nature of the dependence of $X$ on $V$. $\endgroup$
    – Michael Hardy
    Commented Apr 1, 2011 at 0:05
  • $\begingroup$ Sorry about your confusion. Please let me clarify: 1. there exists a well-defined function $f(.)$ so that $x=f(V|\theta)$. Therefore if the pdf of V is known, the pdf of x is known. 2. The pdf of $V$ cannot be uniquely determined by the pdf of $x$ and function $f(.)$. 3. $x=f(V|\theta)$ is set up that, given enough pairs of x and V, $\theta$ can be uniquely identified Does that help? $\endgroup$
    – Haining Yu
    Commented Apr 1, 2011 at 0:41
  • $\begingroup$ I don't think you've got a well-defined math problem here. $V$ could be a vector of 100 independent normally distributed random variables with uniform distributions on the interval $(0,1)$ and $X$ could be the sum of $\theta\in\mathbb{R}$ and a suitable function of the 43rd one of those uniform RVs, so that you get a normally distributed RV with expectation $\theta$. Or you could do all sorts of things like that. At best, you'd have to say more than what you've said to specify a mathematical problem. $\endgroup$
    – Michael Hardy
    Commented Apr 1, 2011 at 22:00
1
$\begingroup$

You definitely would run into identification issues. Suppose $f(x|\mid\theta) = x \theta$. Then multiplying $\theta$ and $\sigma$ by any common factor would result in an identical situation. In this case the MLE would not really be defined.

Generally, as $\theta$ changes, $X$ is revealing a different facet of $V$. For example, if $f$ is a projection map and $\theta$ determines the angle of projection, then $f(V\mid\theta)$ is the shadow of $V$ in the direction of $\theta$. Many of the shadows could be normally distributed, and could be essentially unrelated to each other.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the comment. I see there are possibility for multiple identification issues. First on the scaling issue: in my real problem I have ways to prevent $\mu$ and $\sigma$ from cancelling each other. In fact I can set $\mu$ to a constant. I am indeed more concerned by the relationship between $X$ and $V$. For example, assume $ X = sum_{k}{v_k\theta_k} + c $ where $c$ is a non-zero constant. When I have a large (or infinite) number of samples, $\theta$ should be unique identified? $\endgroup$
    – Haining Yu
    Commented Mar 31, 2011 at 20:48
  • $\begingroup$ Suppose that the V are normally distributed (not necessarily independent). Then for any choice of $\theta_k$, the resulting $X$ is normally distributed with mean and variance depending on $\theta$. There is no such thing as the "correct" $\theta$. $\endgroup$
    – David Harris
    Commented Apr 1, 2011 at 13:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .