MathOverflow is a question and answer site for professional mathematicians. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Consider a random variable $F$ with a distribution parameterized by $\theta$ and another random variable $G$ with a distribution parameterized by a variate of $F$, denoted $f$. Note that $F$ is dependent on $\theta$, but $G$ is dependent on $\theta$ only through $f$.

Consider the likelihood for $\theta$ given observations for $F$ and $G$. $g$ contains no more information about $\theta$ than $f$ (as $G$ is parameterized solely by $f$, not $\theta$), so one would expect

$Likelihood(\theta|f,g) = Likelihood(\theta|f)$.

However, given the usual definition of likelihood, $L(\phi|x)=Pr[X=x|\phi]$, the equality does not hold:

$L(\theta|f,g) = Pr[F=f,G=g|\theta]$,

$L(\theta|f) = Pr[F=f|\theta]$,

$Pr[F=f,G=g|\theta] \neq Pr[F=f|\theta]$.

Is there a notion of likelihood that captures the intuition that the likelihood of a parameter value should not change upon the addition of observations that do not hold additional information about the parameter? I'm not sure what terms to search for or what books and literature to peruse.

share|cite|improve this question
up vote 4 down vote accepted

What you are missing is normalization: your likelihood functions $L(\theta|f,g)$ and $L(\theta|f)$ shouldn't be comparable, because adding up the likelihood for all the possible $\theta$s gives different totals. If you instead define
$\displaystyle L(\phi|x) = \frac{Pr[X=x|\phi]}{\sum_{\phi'} P[X=x|\phi']}$,
then you obtain equalities:

$\displaystyle L(\theta|f,g) = \frac{Pr[F=f, G=g|\theta]}{\sum_{\theta'} P[F=f, G=g|\theta']} = \frac{Pr[G=g|F=f]Pr[F=f|\theta]}{\sum_{\theta'} P[G=g|F=f]P[F=f|\theta']}$

$\displaystyle = \frac{Pr[F=f|\theta]}{\sum_{\theta'} P[F=f|\theta']} = L(\theta|f) $

To learn more, try searching for Bayesian Networks, especially "parameter learning."

share|cite|improve this answer

What you are missing is the concept of a sufficient statistic:

You have $P(F=f | \theta) = a(f|\theta)$, and $P(G=g|F=f)=b(g|f)$ the last one not depending on $\theta$. The distribution of your data $(F,G)$ is given by \begin{equation} P(F=f, G=g|\theta)=a(f|\theta) \cdot b(g|f) \end{equation} and the factorization theorem (see wikipedia article above) says directly that $F$ is a sufficient statistic for $\theta$, based on the sample $(F,G)$. Alternatively, we can use the definition directly: and calculate: \begin{equation} P(F=f,G=g| F=f,\theta)=\frac{P(F=f,G=g|\theta)}{P(F=f|\theta)}=b(g|f) \end{equation} which indeed do not depend on $\theta$. So, you do not need to consider the likelihood based on the full sample $(F,G)$, since you can always do as well using only the sufficient statistic.

An alternative answer is to point out that likelihood functions are not densities, they are only defined up to a multiplicative constant. So any two likelihood functions which have a ratio being independent of $\theta$, are equivalent in the sense that they will lead to equivalent inferences, when inference is based on the likelihood function. (The ratio in question in our case is of course $b(g|f)$.)

share|cite|improve this answer
Thanks, this answer is also very helpful. I was (stupidly) missing that likelihoods are only defined up to a multiplicative constant (of course this is reasonable, given their meaning and use), but the sufficient statistic concept is good to know. – David B. Sep 1 '12 at 21:45

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.