In maximum likelihood estimation, one typically needs to compute the log (natural log) of probability values. When a probability, say $p(x)$, becomes so close to zero, $log(p(x))$ returns -Inf. What is the usual trick to avoid these cases?
$\begingroup$
$\endgroup$
1
1 Answer
$\begingroup$
$\endgroup$
6
Work with the logs of probability directly, rather than trying to compute the probability and then compute the log. You can do arithmetic with the logs, as well; multiplication becomes addition, of course. Addition is somewhat more complicated, but it's not too hard to work out how to do it without taking the exponential back.
In response to JM's question, it's easy to come up with practical problems where the probability of something happening is indeed so low that it underflows a float or double.
answered Oct 23, 2010 at 13:59
-
$\begingroup$ I was curious because my experience with nonconverging iterations as presented to me mostly turned out to be the user providing piss-poor initial estimates. That being said, I see your point. Thanks! $\endgroup$ Commented Oct 23, 2010 at 14:24
-
$\begingroup$ I am already working directly with logs of probability. But still I have this problem. Maybe I should consider those points which cause this problem as outliers and ignore them. Or somebody suggested that I add a very small constant to the probability values (this is possible if you are computing p(x)'s first.) $\endgroup$– eakbasCommented Oct 24, 2010 at 5:33
-
1$\begingroup$ Can you say more about the setting? Your initial wording suggests that you compute p(x), and then take the log, which is the wrong thing to do. Rather, just find log(p(x)) directly. Eg, if x is a sequence of events and p(x) is the product of their individual probabilities, compute the logs of each of those individual probabilities (which should safely avoid underflow), and then add the logs. But maybe you're doing something else? $\endgroup$ Commented Oct 24, 2010 at 5:46
-
$\begingroup$ My bad, I was NOT directly using logs. Thanks, Dylan. $\endgroup$– eakbasCommented Oct 27, 2010 at 2:47
-
$\begingroup$ What if $p(x)$ is a mixture model, e.g. a Gaussian mixture: $p(x;\theta) = \sum_{i=1}^N \pi_i f(x;\mu_i, \Sigma_i)$? You cannot express $log(p(x))$ analytically, and yet you might have some of the $p(\cdot)$ values very close to zero. $\endgroup$– eakbasCommented Oct 31, 2010 at 17:40
-Infunless your starting values for your iteration are really bad... $\endgroup$