Why isn't Likelihood a Probability Density Function? - MathOverflow most recent 30 from http://mathoverflow.net 2013-05-21T12:55:52Z http://mathoverflow.net/feeds/question/10971 http://www.creativecommons.org/licenses/by-nc/2.5/rdf http://mathoverflow.net/questions/10971/why-isnt-likelihood-a-probability-density-function Why isn't Likelihood a Probability Density Function? brabster 2010-01-06T21:53:37Z 2012-06-28T18:25:51Z <p>Hi everyone, first post here...</p> <p>I've been trying to get my head around why a likelihood isn't a probability density function. My understanding says that for an event X and a model parameter m:</p> <p>P(X|m) is a probability density function</p> <p>P(m|X) is not</p> <p>It feels like it should be, and I can't find a clear explanation of why it's not. Does it also mean that a Likelihood can take a value greater than 1?</p> <p>Sorry if this is a stupid question!</p> http://mathoverflow.net/questions/10971/why-isnt-likelihood-a-probability-density-function/10978#10978 Answer by sheldon-cooper for Why isn't Likelihood a Probability Density Function? sheldon-cooper 2010-01-06T22:47:07Z 2010-01-06T22:47:07Z <p>If $X$ is data and $m$ are the parameters, then the likelihood function $l(m) = p(X | m)$. I.e. it's $p(X | m)$, considered as a function of $m$. </p> <p>Both $p(X|m)$ and $p(m|X)$ are pdfs: $p(X|m)$ is a density on $X$ and $p(m|X)$ is a density on $m$. But the likelihood is $p(X|m)$, not as a function of $X$ (it would indeed be a density as a function of X), but as a function of m. So it's not a pdf; in particular, it's not necessarily true that $$\sum_m p(X|m) = 1.$$</p> <p>Edit: just to clarify, $p(m|X)$ isn't the likelihood. $p(X|m)$ is.</p> http://mathoverflow.net/questions/10971/why-isnt-likelihood-a-probability-density-function/10979#10979 Answer by John D. Cook for Why isn't Likelihood a Probability Density Function? John D. Cook 2010-01-06T22:52:21Z 2010-01-06T22:52:21Z <p>From a Bayesian perspective, the reason the likelihood function isn't a probability density is that you haven't multiplied by a prior yet. But once you multiply by a prior distribution, the product is (proportional to) the posterior probability density for the parameters.</p> http://mathoverflow.net/questions/10971/why-isnt-likelihood-a-probability-density-function/11128#11128 Answer by Matus Telgarsky for Why isn't Likelihood a Probability Density Function? Matus Telgarsky 2010-01-08T09:44:02Z 2010-01-08T09:50:37Z <p>A few questions were asked, so a few answers will be given. (main point: likelihood is not necessarily a product density, though this is the common interpretation.)</p> <p>Frequently, the likelihood is the product of densities over some provided set of examples. The examples are drawn i.i.d., and therefore this product density is the density for the corresponding product measure over the product space. What I'm saying is that yes, from this perspective, you have constructed a product density.</p> <p>Since you are dealing with densities, not probabilities, values are not constrained to [0,1], and your density can easily be greater than one. In fact, if you are dealing with dirac measure (which puts all mass on one point on the real line), you essentially have "infinite" density. I put that in quotes since this is not a continuous probability measure, ie it does not have a density wrt to Lebesgue measure, let alone one with infinite mass on a point. (A quick fact check: the corresponding integral wrt lebesgue measure would have value zero since it is off zero only on a set of lebesgue measure zero, which means it is not a probability distribution; but it was, which contradicts this being its density.) perhaps a more apt example: any (continuous) distribution on [0,0.5] will have to have density greater than one on a set of nonzero lebesgue measure. (you can try to construct a sequence of these which convergence to something which violates what i said, but that will be the density of something which is not continuous!)</p> <p>things can get a little confusing because you can write discrete probability distributions as densities wrt a measure putting 1 on each point in the support set of the probability (ie it is counting measure wrt that set). NOTE that this is a density wrt a measure which is NOT a probability measure. But anyway, the density values at each point are exactly the probability values. This allows an interchanging probability masses and densities, which can be confusing.</p> <p>I'll close with some further reading. A good book on machine learning is "A probabilistic Theory of Pattern Recognition" by Devroye, Gyorfi, Lugosi. Chapter 15 is on maximum likelihood and you'll notice they do NOT define likelihood as being a product probability or density, but rather as a product of functions. This is because they are careful to encompass the differing interpretations; rather, they ignore the interpretations there and work out the math.</p> http://mathoverflow.net/questions/10971/why-isnt-likelihood-a-probability-density-function/30822#30822 Answer by Myke for Why isn't Likelihood a Probability Density Function? Myke 2010-07-06T21:44:27Z 2010-07-06T21:44:27Z <p>If m is a model parameter then it is not a RV. So it does not have a probability density function. m is just an unknown paramter.</p> http://mathoverflow.net/questions/10971/why-isnt-likelihood-a-probability-density-function/65495#65495 Answer by etc-etc for Why isn't Likelihood a Probability Density Function? etc-etc 2011-05-20T02:55:05Z 2011-05-20T02:55:05Z <p>You are essentially asking why likelihood density function cannot be used as probability density function. For a non-informative (uniform) Bayesian prior, the likelihood density function divided by its integral over entire space will give the probability density function. It is the normalization that makes the two different. </p> http://mathoverflow.net/questions/10971/why-isnt-likelihood-a-probability-density-function/99491#99491 Answer by Sepideh for Why isn't Likelihood a Probability Density Function? Sepideh 2012-06-13T19:49:03Z 2012-06-13T19:49:03Z <p>Likelihood is the chance that the reality you've hypothesized could have produced the particular data you got.</p> <p>Likelihood: The probability of data given a hypothesis.</p> <p>However Probability is the chance that the reality you're considering is true, given the data you have.</p> <p>Bayesian Probability: Probability of a hypothesis given data </p> http://mathoverflow.net/questions/10971/why-isnt-likelihood-a-probability-density-function/100887#100887 Answer by Stéphane Laurent for Why isn't Likelihood a Probability Density Function? Stéphane Laurent 2012-06-28T18:25:51Z 2012-06-28T18:25:51Z <p>The accepted answer is wrong. The likelihood is the function $\theta \mapsto L(\theta \mid x)=f(x \mid \theta)$ for a given $x$ in the observations space and $f(\cdot \mid \theta)$ is a Radon-Nikodym derivative of $P_\theta$ when the statistical model is given by a family of probabilities ${(P_\theta)}_{\theta\in\Theta}$ on the observations space.</p> <p>In general there's not even a $\sigma$-field in the parameter space $\Theta$, hence the question <em>"is the likelihood a pdf ?"</em> has not even a sense!</p> <p>For more information see <a href="http://stats.stackexchange.com/questions/31238/what-is-the-reason-that-a-likelihood-function-is-not-a-pdf" rel="nofollow">http://stats.stackexchange.com/questions/31238/what-is-the-reason-that-a-likelihood-function-is-not-a-pdf</a> and <a href="http://stats.stackexchange.com/questions/29682/how-to-rigorously-define-the-likelihood" rel="nofollow">http://stats.stackexchange.com/questions/29682/how-to-rigorously-define-the-likelihood</a></p>