Let $P$ be a probability measure and $f$ be some probability density function (not necessarily related to $P$). Consider the function $$ L(X_1,\ldots,X_n) =\frac1n\sum_{i=1}^n\log f(X_i), \quad X_i\overset{iid}{\sim}P. $$

This is just the empirical log-likelihood from statistics. Are there any concentration inequalities that are specialized for $L(X_1,\ldots,X_n)$?

Surprisingly, I could not find any in the literature besides the generic ones. That is, we can naively apply generic inequalities such as Bernstein, etc. or assume that $\log f(X_i)$ is a subgaussian random variable, but this is a bit unsatisfying. For example, in the simplest case where $f$ is a Gaussian density, $\log f(x)$ is a quadratic function, and hence not subgaussian if $X_i$ is also Gaussian.

It seems like it should be possible to exploit the fact that $f$ is a density to come up with something more useful.


Assume $X$ has density $f$ and put $Y=\log f(X)$. Then, to compute the moment-generating function of $Y$, we write $$ E e^{\lambda Y} =E e^{\lambda\log f(X)} =E[f(X)^{\lambda}] =\int_{-\infty}^\infty f^{\lambda+1}(x)dx. $$ For $\lambda\ge0$, the latter integral is finite for all densities, because the region where $f\ge1$ must have (Lebesgue) measure $\le 1$, and elsewhere $f<1$ so raising it to a power $>1$ only decreases the value. So the answer depends on your ability to analytically control $F(\lambda):=\int_{-\infty}^\infty f^{\lambda+1}(x)dx$. In the normal Gaussian case, we have $F(\lambda)=1/\sqrt{\lambda+1}$. You can then apply the standard exponential bounding technique (plus Markov's inequality) to the sum of $Y$'s.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.