Consider a random variable $F$ with a distribution parameterized by $\theta$ and another random variable $G$ with a distribution parameterized by a variate of $F$, denoted $f$. Note that $F$ is dependent on $\theta$, but $G$ is dependent on $\theta$ only through $f$.

Consider the likelihood for $\theta$ given observations for $F$ and $G$. $g$ contains no more information about $\theta$ than $f$ (as $G$ is parameterized solely by $f$, not $\theta$), so one would expect

$Likelihood(\theta|f,g) = Likelihood(\theta|f)$.

However, given the usual definition of likelihood, $L(\phi|x)=Pr[X=x|\phi]$, the equality does not hold:

$L(\theta|f,g) = Pr[F=f,G=g|\theta]$,

$L(\theta|f) = Pr[F=f|\theta]$,

$Pr[F=f,G=g|\theta] \neq Pr[F=f|\theta]$.

Is there a notion of likelihood that captures the intuition that the likelihood of a parameter value should not change upon the addition of observations that do not hold additional information about the parameter? I'm not sure what terms to search for or what books and literature to peruse.

| cite | improve this question | | | | |

What you are missing is normalization: your likelihood functions $L(\theta|f,g)$ and $L(\theta|f)$ shouldn't be comparable, because adding up the likelihood for all the possible $\theta$s gives different totals. If you instead define
$\displaystyle L(\phi|x) = \frac{Pr[X=x|\phi]}{\sum_{\phi'} P[X=x|\phi']}$,
then you obtain equalities:

$\displaystyle L(\theta|f,g) = \frac{Pr[F=f, G=g|\theta]}{\sum_{\theta'} P[F=f, G=g|\theta']} = \frac{Pr[G=g|F=f]Pr[F=f|\theta]}{\sum_{\theta'} P[G=g|F=f]P[F=f|\theta']}$

$\displaystyle = \frac{Pr[F=f|\theta]}{\sum_{\theta'} P[F=f|\theta']} = L(\theta|f) $

To learn more, try searching for Bayesian Networks, especially "parameter learning."

| cite | improve this answer | | | | |

What you are missing is the concept of a sufficient statistic:


You have $P(F=f | \theta) = a(f|\theta)$, and $P(G=g|F=f)=b(g|f)$ the last one not depending on $\theta$. The distribution of your data $(F,G)$ is given by \begin{equation} P(F=f, G=g|\theta)=a(f|\theta) \cdot b(g|f) \end{equation} and the factorization theorem (see wikipedia article above) says directly that $F$ is a sufficient statistic for $\theta$, based on the sample $(F,G)$. Alternatively, we can use the definition directly: and calculate: \begin{equation} P(F=f,G=g| F=f,\theta)=\frac{P(F=f,G=g|\theta)}{P(F=f|\theta)}=b(g|f) \end{equation} which indeed do not depend on $\theta$. So, you do not need to consider the likelihood based on the full sample $(F,G)$, since you can always do as well using only the sufficient statistic.

An alternative answer is to point out that likelihood functions are not densities, they are only defined up to a multiplicative constant. So any two likelihood functions which have a ratio being independent of $\theta$, are equivalent in the sense that they will lead to equivalent inferences, when inference is based on the likelihood function. (The ratio in question in our case is of course $b(g|f)$.)

| cite | improve this answer | | | | |
  • $\begingroup$ Thanks, this answer is also very helpful. I was (stupidly) missing that likelihoods are only defined up to a multiplicative constant (of course this is reasonable, given their meaning and use), but the sufficient statistic concept is good to know. $\endgroup$ – David B. Sep 1 '12 at 21:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.