Welcome to MathOverflow

Question

2

1

In maximum likelihood estimation, one typically needs to compute the log (natural log) of probability values. When a probability, say $p(x)$, becomes so close to zero, $log(p(x))$ returns -Inf. What is the usual trick to avoid these cases?

edited Oct 27 2010 at 7:17

Andrey Rekalo
13.9k●1●32●68

asked Oct 21 2010 at 4:34

eakbas
121●6

1

If you're maximizing, I can't see how you'd hit a -Inf unless your starting values for your iteration are really bad... – J. M. Oct 21 2010 at 5:29

Dylan Thurston · Accepted Answer · 2010-10-23 14:28:03Z UTC

5

Work with the logs of probability directly, rather than trying to compute the probability and then compute the log. You can do arithmetic with the logs, as well; multiplication becomes addition, of course. Addition is somewhat more complicated, but it's not too hard to work out how to do it without taking the exponential back.

In response to JM's question, it's easy to come up with practical problems where the probability of something happening is indeed so low that it underflows a float or double.

edited Oct 23 2010 at 14:28

answered Oct 23 2010 at 13:59

Dylan Thurston
3,544●10●22

I was curious because my experience with nonconverging iterations as presented to me mostly turned out to be the user providing piss-poor initial estimates. That being said, I see your point. Thanks! – J. M. Oct 23 2010 at 14:24

I am already working directly with logs of probability. But still I have this problem. Maybe I should consider those points which cause this problem as outliers and ignore them. Or somebody suggested that I add a very small constant to the probability values (this is possible if you are computing p(x)'s first.) – eakbas Oct 24 2010 at 5:33

1

Can you say more about the setting? Your initial wording suggests that you compute p(x), and then take the log, which is the wrong thing to do. Rather, just find log(p(x)) directly. Eg, if x is a sequence of events and p(x) is the product of their individual probabilities, compute the logs of each of those individual probabilities (which should safely avoid underflow), and then add the logs. But maybe you're doing something else? – Dylan Thurston Oct 24 2010 at 5:46

My bad, I was NOT directly using logs. Thanks, Dylan. – eakbas Oct 27 2010 at 2:47

What if $p(x)$ is a mixture model, e.g. a Gaussian mixture: $p(x;\theta) = \sum_{i=1}^N \pi_i f(x;\mu_i, \Sigma_i)$? You cannot express $log(p(x))$ analytically, and yet you might have some of the $p(\cdot)$ values very close to zero. – eakbas Oct 31 2010 at 17:40

show 2 more comments

Numeric problem when evaluating log of a pdf

Remember to vote up questions/answers you find interesting or helpful (requires 15 reputation points)

1 Answer

You can accept an answer to one of your own questions by clicking the check mark next to it. This awards 15 reputation points to the person who answered and 2 reputation points to you.