## Numeric problem when evaluating log of a pdf

In maximum likelihood estimation, one typically needs to compute the log (natural log) of probability values. When a probability, say $p(x)$, becomes so close to zero, $log(p(x))$ returns -Inf. What is the usual trick to avoid these cases?

-
If you're maximizing, I can't see how you'd hit a -Inf unless your starting values for your iteration are really bad... – J. M. Oct 21 2010 at 5:29

Work with the logs of probability directly, rather than trying to compute the probability and then compute the log. You can do arithmetic with the logs, as well; multiplication becomes addition, of course. Addition is somewhat more complicated, but it's not too hard to work out how to do it without taking the exponential back.

In response to JM's question, it's easy to come up with practical problems where the probability of something happening is indeed so low that it underflows a float or double.

-
I was curious because my experience with nonconverging iterations as presented to me mostly turned out to be the user providing piss-poor initial estimates. That being said, I see your point. Thanks! – J. M. Oct 23 2010 at 14:24
I am already working directly with logs of probability. But still I have this problem. Maybe I should consider those points which cause this problem as outliers and ignore them. Or somebody suggested that I add a very small constant to the probability values (this is possible if you are computing p(x)'s first.) – eakbas Oct 24 2010 at 5:33
Can you say more about the setting? Your initial wording suggests that you compute p(x), and then take the log, which is the wrong thing to do. Rather, just find log(p(x)) directly. Eg, if x is a sequence of events and p(x) is the product of their individual probabilities, compute the logs of each of those individual probabilities (which should safely avoid underflow), and then add the logs. But maybe you're doing something else? – Dylan Thurston Oct 24 2010 at 5:46
My bad, I was NOT directly using logs. Thanks, Dylan. – eakbas Oct 27 2010 at 2:47
What if $p(x)$ is a mixture model, e.g. a Gaussian mixture: $p(x;\theta) = \sum_{i=1}^N \pi_i f(x;\mu_i, \Sigma_i)$? You cannot express $log(p(x))$ analytically, and yet you might have some of the $p(\cdot)$ values very close to zero. – eakbas Oct 31 2010 at 17:40