Take the 2-minute tour ×
MathOverflow is a question and answer site for professional mathematicians. It's 100% free, no registration required.

In maximum likelihood estimation, one typically needs to compute the log (natural log) of probability values. When a probability, say $p(x)$, becomes so close to zero, $log(p(x))$ returns -Inf. What is the usual trick to avoid these cases?

share|improve this question
1  
If you're maximizing, I can't see how you'd hit a -Inf unless your starting values for your iteration are really bad... –  J. M. Oct 21 '10 at 5:29

1 Answer 1

up vote 5 down vote accepted

Work with the logs of probability directly, rather than trying to compute the probability and then compute the log. You can do arithmetic with the logs, as well; multiplication becomes addition, of course. Addition is somewhat more complicated, but it's not too hard to work out how to do it without taking the exponential back.

In response to JM's question, it's easy to come up with practical problems where the probability of something happening is indeed so low that it underflows a float or double.

share|improve this answer
    
I was curious because my experience with nonconverging iterations as presented to me mostly turned out to be the user providing piss-poor initial estimates. That being said, I see your point. Thanks! –  J. M. Oct 23 '10 at 14:24
    
I am already working directly with logs of probability. But still I have this problem. Maybe I should consider those points which cause this problem as outliers and ignore them. Or somebody suggested that I add a very small constant to the probability values (this is possible if you are computing p(x)'s first.) –  eakbas Oct 24 '10 at 5:33
1  
Can you say more about the setting? Your initial wording suggests that you compute p(x), and then take the log, which is the wrong thing to do. Rather, just find log(p(x)) directly. Eg, if x is a sequence of events and p(x) is the product of their individual probabilities, compute the logs of each of those individual probabilities (which should safely avoid underflow), and then add the logs. But maybe you're doing something else? –  Dylan Thurston Oct 24 '10 at 5:46
    
My bad, I was NOT directly using logs. Thanks, Dylan. –  eakbas Oct 27 '10 at 2:47
    
What if $p(x)$ is a mixture model, e.g. a Gaussian mixture: $p(x;\theta) = \sum_{i=1}^N \pi_i f(x;\mu_i, \Sigma_i)$? You cannot express $log(p(x))$ analytically, and yet you might have some of the $p(\cdot)$ values very close to zero. –  eakbas Oct 31 '10 at 17:40

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.