Is there a notion of likelihood that incorporates information content?

Question

Consider a random variable $F$ with a distribution parameterized by $\theta$ and another random variable $G$ with a distribution parameterized by a variate of $F$, denoted $f$. Note that $F$ is dependent on $\theta$, but $G$ is dependent on $\theta$ only through $f$.

Consider the likelihood for $\theta$ given observations for $F$ and $G$. $g$ contains no more information about $\theta$ than $f$ (as $G$ is parameterized solely by $f$, not $\theta$), so one would expect

$Likelihood(\theta|f,g) = Likelihood(\theta|f)$.

However, given the usual definition of likelihood, $L(\phi|x)=Pr[X=x|\phi]$, the equality does not hold:

$L(\theta|f,g) = Pr[F=f,G=g|\theta]$,

$L(\theta|f) = Pr[F=f|\theta]$,

$Pr[F=f,G=g|\theta] \neq Pr[F=f|\theta]$.

Is there a notion of likelihood that captures the intuition that the likelihood of a parameter value should not change upon the addition of observations that do not hold additional information about the parameter? I'm not sure what terms to search for or what books and literature to peruse.

Answer 1

What you are missing is normalization: your likelihood functions $L(\theta|f,g)$ and $L(\theta|f)$ shouldn't be comparable, because adding up the likelihood for all the possible $\theta$s gives different totals. If you instead define
$\displaystyle L(\phi|x) = \frac{Pr[X=x|\phi]}{\sum_{\phi'} P[X=x|\phi']}$,
then you obtain equalities:

$\displaystyle L(\theta|f,g) = \frac{Pr[F=f, G=g|\theta]}{\sum_{\theta'} P[F=f, G=g|\theta']} = \frac{Pr[G=g|F=f]Pr[F=f|\theta]}{\sum_{\theta'} P[G=g|F=f]P[F=f|\theta']}$

$\displaystyle = \frac{Pr[F=f|\theta]}{\sum_{\theta'} P[F=f|\theta']} = L(\theta|f) $

To learn more, try searching for Bayesian Networks, especially "parameter learning."

Answer 2

What you are missing is the concept of a sufficient statistic:

http://en.wikipedia.org/wiki/Sufficient_statistic

You have $P(F=f | \theta) = a(f|\theta)$, and $P(G=g|F=f)=b(g|f)$ the last one not depending on $\theta$. The distribution of your data $(F,G)$ is given by \begin{equation} P(F=f, G=g|\theta)=a(f|\theta) \cdot b(g|f) \end{equation} and the factorization theorem (see wikipedia article above) says directly that $F$ is a sufficient statistic for $\theta$, based on the sample $(F,G)$. Alternatively, we can use the definition directly: and calculate: \begin{equation} P(F=f,G=g| F=f,\theta)=\frac{P(F=f,G=g|\theta)}{P(F=f|\theta)}=b(g|f) \end{equation} which indeed do not depend on $\theta$. So, you do not need to consider the likelihood based on the full sample $(F,G)$, since you can always do as well using only the sufficient statistic.

An alternative answer is to point out that likelihood functions are not densities, they are only defined up to a multiplicative constant. So any two likelihood functions which have a ratio being independent of $\theta$, are equivalent in the sense that they will lead to equivalent inferences, when inference is based on the likelihood function. (The ratio in question in our case is of course $b(g|f)$.)