Fisher information with vanishing probability

Question

I am confused about the definition of the Fisher information and the case when probability is 0. Consider discrete set $\epsilon$ of possible measurement outcomes. Fisher information is defined as: $$F(\theta) = \sum\limits_{\epsilon} \frac{1}{p(\epsilon|\theta)}\left( \frac{\partial p(\epsilon|\theta)}{\partial \theta}\right)^2.$$

What happens when in the sum one of the probabilities $p(\epsilon_0|\theta)=0$. I can imagine 2 situations:

1. $p(\epsilon_0|\theta) = 0$ and $\frac{\partial p(\epsilon_0|\theta)}{\partial \theta} \neq 0$. Then $F(\theta) = \infty$.
2. $p(\epsilon_0|\theta) = 0$ and $\frac{\partial p(\epsilon_0|\theta)}{\partial \theta} = 0$. Then we have $0/0$ which might be determined from L'Hospital rule and give finite value. Then $$\frac{1}{p(\epsilon_0|\theta)}\left( \frac{\partial p(\epsilon_0|\theta)}{\partial \theta}\right)^2 = 2 \frac{\partial^2 p(\epsilon_0|\theta)}{\partial \theta^2}.$$

Does it really make sense to take into account measurement outcome $\epsilon_0$ for which $p(\epsilon_0|\theta) = 0$ based on the fact that it does not contribute to the variance of the estimator?

Answer 1

An infinite Fisher information $F(\theta)$ means that the lower bound $1/F(\theta)$ in the error of an unbiased estimator of $\theta$ is zero. A simple example is the Bernoulli trial, success or failure with probabilities $\theta$ and $1-\theta$. The Fisher information is $$F(\theta)= \frac{1}{\theta}\frac{1}{1-\theta}$$ So $F(\theta)$ diverges for $\theta=0$ or $1$, which makes sense because then there are no fluctuations in the outcome so there is no limit to the precision with which you can predict it.